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LITERATURE  REVIEW:  COST  AND  TRAINING  EFFECTIVENESS 

INTRODUCTION 

This  report  is  one  in  a  series  of  documents  dealing  with  cost 
and  training  effectiveness  of  the  Training  Extension  Course  (TEC) 
Program.  This  document  reports  on  a  review  and  evaluation  of 
literature  with  the  objective  of  identifying  State-of-the-Art  methods 
and  techniques  applicable,  in  whole  or  part,  to  the  performance  of 
Cost  and  Training  Effectiveness  Analyses  (CTEA).1  The  review  in¬ 
cluded  literature  from  all  available  military  and  non-military 
sources.  Specifically,  the  purpose  of  the  review  was  to  identify 
those  approaches  which  have  been  productive  with  respect  to  train¬ 
ing  systems  evaluations  and  to  identify  where  discrepancies  in 
methods  and  techniques  exist.  The  scope  of  this  review  is  limited 
to  that  literature  having  some  bearing  on  methodology  useful  to 
the  evaluation  of  TEC. 

A  following  document  will  describe  the  methodology  formulated, 
in  light  of  this  review,  to  carry  through  a  TEC  CTEA.  It  also 
will  contain  an  assessment  of  that  methodology  with  regard  to  its 
strong  and  weak  points.  Finally,  that  document  will  describe  ap¬ 
plication  of  the  methodology  to  a  set  of  hypothetical  cost  and 
training  data.  Thus,  this  present  document  can  be  viewed  as  back¬ 
ground  and  introduction  to  that  document. 


CTEA  CONCEPT 

TEC  was  designed  as  a  series  of  multimedia,  exportable,  per¬ 
formance-oriented  training  packages.  Its  purpose  is  to  provide 
high  quality,  uniform  instruction  in  critical  skills  to  individual 
and  small  groups  of  soldiers  (Litton  Systems,  Inc.,  1978). 
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Throughout  this  report  the  phrases  CTEA,  evaluation  of  TEC, 
and  training  (program)  evaluation  are  used.  These  phrases  are  not 
synonymous,  though  they  are  closely  related.  CTEA  refers  to  an 
evaluation  of  a  training  program  using  methodology  consistent  with 
the  Army's,  as  set  forth  in  the  TRASANA  (U.  S.  Army  TRADOC  Systems 
Analysis  Activity)  CTEA  Handbook.  Evaluation  of  TEC  refers,  in 
general,  to  any  assessment  (not  necessarily  consistent  with  Army 
CTEA  methodology)  of  the  TEC  program.  In  context,  however,  ws 
always  accept  that  an  evaluation  of  TEC  will  be  consistent  with 
CTEA  methodology.  Training  (program)  evaluation  refers  to  any 
methodology.  Training  (program)  evaluation  refers  to  any  assess¬ 
ment  of  any  type  of  any  training  program.  CTEA  is  a  special  case 
of  training  (program)  evaluation,  and  TEC  CTEA  is  a  special  case 
of  evaluation  of  TEC. 


GENERAL  REQUIREMENTS 


A  TEC  CTEA  must  involve,  as  does  any  CTEA,  a  comparison  of 
the  effectiveness  and  costs  of  alternative  training  systems.  In 
addition  to  examining  a  program  of  instruction  (such  as  TEC),  a 
CTEA  can  also  examine  training  concepts,  training  equipment,  train¬ 
ing  strategies,  training  organization  and  facilities,  and  training 
impacts  of  new  material,  organizations  or  tactics  (TRASANA,  1976). 
Ultimately,  any  decision  regarding  a  modification  of  any  part  of 
the  training  system  must  be  made  in  consideration  of  the  effects 
of  that  modification  on  the  effectiveness  of  the  force  structure. 
Thus,  a  CTEA  must  relate  a  (proposed  or  actual)  change  in  the 
training  system  to  its  ultimate  impact  on  combat  effectiveness. 
There  should  be  a  clear  "audit  trail"  from  the  former  to  the  lat¬ 
ter.  This  means  the  CTEA  must  be  performance-oriented.  There 
are  three  basic  types  of  CTEA:  the  Train-up  Study  (TUS),  the 
Training  Analysis  for  COEA2  (TAC) ,  and  the  Training  Development 
Study  (TDS) .  This  final  type  of  CTEA  is  conducted  to  evaluate 
and  compare  training  effectiveness,  resource  impacts,  costs,  and 
cost  benefits  of  alternative  training  systems  proposed  to  correct 
training  deficiencies,  to  facilitate  attainment  of  individual  and 
collective  performance  objectives,  or  to  achieve  training  economies 
(TRASANA,  1976).  The  evaluation  of  TEC  is  a  TDS-type  CTEA. 

In  any  CTEA,  the  proposed  alternative  must  be  compared  with 
the  baseline  system,  which  is  the  best  currently  available  train¬ 
ing  system.  The  comparison  can  be  of  three  different  types: 
fixed  cost,  variable  effectiveness;  variable  costs,  fixed  effec¬ 
tiveness;  or  variable  costs,  variable  effectiveness.  In  the 
TEC  case,  the  comparison  is  of  the  third  variety. 

Because  TEC  is  designed  to  complement,  or  enhance,  existing 
training  systems;  and  not  (necessarily)  to  substitute  for  any 
part  of  the  extant  training  system;  a  TEC  CTEA  must  reflect  that 
complementary  nature.  This  means  we  must  explicitly  recognize 
that,  a  purpose  of  TEC  being  to  enhance  effectiveness,  a  "fixed 
effectiveness"  type  of  CTEA  methodology  would  be  inappropriate 
for  TEC  evaluation.  At  the  same  time,  notwithstanding  the  fact 
that  minimizing  training  costs  is  an  important  goal,  TEC  is  not 
necessarily  intended  as  a  dol lar-for-dol 1 ar  replacement  for  any 
existing  training  expenditures.  Thus,  a  "fixed  cost"  type  of 
CTEA  methodology  for  TEC  would  also  be  inappropriate.  What  re¬ 
mains,  then,  is  that  the  TEC  CTEA  methodology  must  be  a  "variable 
cost,  variable  effectiveness"  type  of  comparison.  Such  a  com¬ 
parison  is  usually  referred  to  as  a  cost  benefit  analysis. 
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Cost  and  Operational  Effectiveness  Analysis. 


In  contrast,  when  either  cost  or  effectiveness  is  held  fixed, 
that  comparison  is  called  a  cost  effectiveness  analysis  (Sassone 
and  Shaffer  ,  1978) . 

The  methodological  requirements  for  a  TEC  CTEA  are  dictated 
by  the  usual  CTEA  considerations  (TRASANA,  1976)  as  well  as  by  the 
TEC  CTEA's  kinship  to  a  cost  benefit  analysis  (Mishan,  1976; 
Sassone  and  Shaffer,  1978).  Five  requirements  can  be  identified: 

o  the  baseline  training  system  must  be  identified  and 
its  parameters  quantified 

o  the  training  system  ’nclusive  of  TEC  must  be  defined 
and  its  parameters  quantified 

o  an  input  cost  function  must  be  developed  which  maps 
any  training  system  configuration  into  its  dollar  cost 

o  an  effectiveness  production  function  must  be  developed 
which  maps  any  training  system  configuration  into  an 
objective  measure  of  its  effectiveness 

o  an  approach  must  be  devised  which  objectively  trans¬ 
lates  changes  in  effectiveness  into  corresponding 
dol ?ar  values. 


PROPOSED  APPROACH  FOR  TEC 

The  methods  of  microeconomics  and  welfare  economics  can  be 
used  to  approach  the  methodological  requirements  for  a  TEC  CTEA. 
The  evaluation  procedures  are  objective,  and  result  in  ultimate 
dollar  valuations  of  the  TEC  training  concept. 

In  the  context  of  military  training,  a  model  can  be  set  up 
as  follows.  A  group  (person,  platoon,  company,  battalion,  etc.) 
has  a  number  of  missions  for  which  it  is  responsible.  There  are 
a  number  of  different  types  of  training  activities  in  which  the 
group  can  engage,  and  each  type  of  training  activity  makes  some 
contribution  to  the  ability  of  the  group  to  carry  out  each  mis¬ 
sion.  To  be  sure,  certain  types  of  training  may  enhance  ability 
in  one  mission  area  more  than  in  another.  Each  training  activity 
is  conducted  in  self-contained  "units"  so  it  is  meaningful  to 
measure  the  amount  of  training  a  group  receives  as  a  number  of 
units  of  each  type  training.  The  overall  ability  and  proficiency 
with  which  the  group  carries  out  its  assigned  missions  is  refer¬ 
red  to  as  the  group  effect i veness ,  and  we  assume  a  reasonable 
objective  measure  of  group  effectiveness  exists.  The  costs  of 
carrying  out  each  type  of  training  are  also  assumed  to  be  known, 
and  the  group's  training  budget  is  assumed  known  and  fixed. 
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Suppose  a  new  training  concept  is  developed  and  is  considered 
for  introduction  into  this  milieu.  A  procedure  is  needed  to  deter¬ 
mine  whether,  and  to  what  extent,  the  new  training  concept  is  eco¬ 
nomically  viable.  The  basic  approach  is  to  determine  whether  the 
present  value  of  the  sum  of  the  net  values  of  the  new  training  to 
each  group  utilizing  that  training  over  the  "lifetime"  of  the  train 
ing  program  exceeds  the  development  cost  of  the  training  program. 
The  key  to  implementing  the  approach  is  the  determination  of  the 
net  value  of  the  new  training  to  a  group  using  it.  This  can  be  ac¬ 
complished  using  either  one  of  two  concepts  from  welfare  economics. 
The  first  is  the  compensating  budget  variation  (Currie,  et.  al., 
1971).  Here  one  attempts  to  determine  the  maximum  training  budget 
reduction  a  group  can  suffer  in  the  presence  of  the  new  training 
program  which  still  permits  the  previous  level  of  effectiveness  - 
the  level  reached  with  the  initial  budget  and  no  new  training  - 
lo  be  achieved.  The  second  concept  is  the  equivalent  budget  varia¬ 
tion.  Here  one  attempts  to  determine  the  minimum  training  budget 
increase  which  a  group  must  receive  to  achieve  the  same  increase 
in  effectiveness  without  the  new  training  program  as  can  be  achieve 
with  the  initial  budget  and  with  the  new  training  program.  Under 
certain  conditions,  the  compensating  budget  variation  and  the 
equivalent  budget  variation  are  numerically  equivalent.  More 
generally,  however,  the  latter  exceeds  the  former.3  Which  of  the 
two  concepts  provides  the  more  appropriate  measure  of  the  value  of 
the  new  training  program  depends  on  the  objective  of  the  organi¬ 
zation.  If  the  objective  is  to  minimize  the  cost  of  achieving  the 
current  level  of  effectiveness,  the  compensating  budget  variation 
is  more  appropriate.  If  the  objective  is  to  increase  effectiveness 
while  maintaining  the  initial  budget  level,  the  equivalent  budget 
variation  is  more  appropriate.  To  the  extent  that  the  organi¬ 
zation's  objective  is  uncertain,  or  to  the  extent  it  combines  both 
mentioned  objectives,  a  weighted  average  of  the  two  variations  can 
be  used. 

The  numerical  values  of  both  the  compensating  and  equivalent 
budget  variations  can  be  derived  from  the  effectiveness  production 
funct ion  and  the  group's  training  budget  constraint  (or  input  cost 
function) .  The  effectiveness  production  function  is  the  relation¬ 
ship  between  the  inputs  and  output  of  the  training  process.4  The 
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The  reason  for  this  asymmetry,  which  was  first  pointed  out 
by  A.  Henderson  ("Consumers  Surplus  and  the  Compensating  Variation 
Review  of  Economic  Studies,  19^0  is  somewhat  technical  and  unim¬ 
portant  for  present  purposes. 

<* 

Here  we  are  using  concepts  borrowed  from  the  theory  of  pro¬ 
duction  in  economics.  See  Henderson  and  Quandt,  1971;  Takayama, 
197^;  Intri 1 igator ,  1971. 


inputs  are  the  various  types  and  amounts  of  training  the  group 
receives,  along  with  the  relevant  attributes  of  the  group  itself 
(such  as  intellignece  and  prior  training);  the  output  is  the  as¬ 
sumed,  objective  and  measurable  characteristic,  effectiveness. 

The  training  budget  constraint  is  the  relationship  among  the  dol¬ 
lar  value  of  the  training  budget,  the  unit  costs  of  the  various 
types  of  available  training,  and  the  quantities  of  each  type  of 
training  which  can  be  afforded. 

The  training  budget  constraint  is  a  linear  function  and 
is  conceptually  straightforward.  The  effectiveness  production 
function,  on  the  other  hand,  is  (in  general)  non-linear  and  is 
rich  in  possible  specifications.  In  considering  the  impact  on 
the  production  function  of  the  introduction  of  a  new  training 
program,  there  appear  to  be  four  distinctly  different  ways  in 
which  the  new  program  can  affect  effectiveness.  First,  the  new 
program  can  influence  effectiveness  in  the  same  way  that  every 
other  program  does,  i.  e.,  it  is  simply  another  among  many  train¬ 
ing  programs.  Second,  the  new  program  can  enhance  or  augment  the 
effect  of  one  or  several  of  the  other  training  programs,  i.  e., 
it  can  make  it  seem  like  more  of  some  other  training  is  being 
received  than  is  actually  the  case.  Third,  the  new  program  can 
uniformly  augment  all  the  other  training,  i.  e.,  it  can  raise 
the  entire  effectiveness  production  function.  Finally,  the  new 
training  program  can  increase  the  subst i tutab i I i ty  of  one  type 
of  training  for  another,  permitting  some  of  a  less  costly  type 
of  training  to  be  substituted  for  some  of  a  more  costly  type  of 
training  while  maintaining  the  initial  level  of  effectiveness. 

The  effectiveness  production  function  must  be  empirically 
determined.  Multiple  regression  analysis  is  the  logical  start¬ 
ing  point  for  this  estimation  process.  The  observations  (data 
points)  may  be  cross-sectional  or  longitudinal,  or  perhaps  a 
pooled  sample.  The  regressand  is  the  effectiveness  measure 
achieved  by  the  group.  The  regressors  would  be  the  amounts  of 
each  type  of  training,  as  well  as  such  standardizing  variables 
as  intelligence  and  prior  training,  relevant  to  each  effective¬ 
ness  observation.  The  training  budget  constraint  need  not  be 
empirically  estimated,  but  can  be  constructed  using  available 
data . 

The  details  and  development  of  a  TEC  CTEA  methodology  based 
on  this  approach  will  be  presented  in  a  subsequent  report. 

LITERATURE  REVIEW 

As  mentioned  at  the  outset,  this  survey  and  discussion  of 
literature  is  motivated  by  the  goal  of  carrying  out  a  Cost  and 


Training  Effectiveness  Analysis  of  the  Army's  Training  Extension 
Course  Program.  CTEA  methodology  is  now  in  the  formative  stage. 

It  was  quite  naturally  hoped  that  a  review  of  rclc.vant  methodol¬ 
ogy  oriented  literature  would  provide  information,  concepts,  and 
techniques  useful  to  the  development  of  a  TEC  CTEA  methodology. 
Thus,  the  objective  of  the  research  effort  reported  here  is  to 
evaluate  recent  literature  with  regard  to  its  applicability  to  the 
TEC  CTEA  problem. 


LITERATURE  SEARCH 

Two  more  or  less  separate  bodies  of  literature  we re  searched 
for  TEC  CTEA  -  relevant  material.  The  first  body  was  the  training 
evaluation  literature.  This  literature,  comprised  primarily  of 
military  documents,  is  problem  oriented.  It  generally  focuses  on 
a  specific  training  program.  In  contrast,  the  second  body  of  liter¬ 
ature  was  primarily  theoretical  in  orientation.  This  was  the  eco¬ 
nomic  evaluation  literature,  comprised  of  monographs  and  journal 
articles  dealing  with  the  general  economic  considerations  of 
evaluating  any  type  of  program  or  project.  Because  of  inherent 
differences  between  these  two  bodies  of  literature,  different 
search  procedures  were  used  in  each  case  to  ferret  out  TEC-rele¬ 
vant  material.  For  the  training  literature,  computerized  litera¬ 
ture  searches  were  employed  to  generate  extensive  lists  of  titles 
and  abstracts.  These  were  screened  on  the  basis  of  apparent  TEC 
relevance,  and  thus  a  set  of  documents  was  chosen  fo*-  detailed 
review.  The  economic  evaluation  literature  was  selected  from  the 
subject  classification  indices  of  the  Journal  of  Economic  Litera¬ 
ture,  from  the  bibliographies  of  recent  comprehensive  monographs 
in  project  evaluation,  and  from  among  the  well-known  "classics" 
of  the  field. 


TRAINING  LITERATURE 

The  approach  adopted  for  the  training  literature  survey  is 
not  entirely  orthodox.  Recognizing  that  a  TEC  CTEA  would  in¬ 
volve  numerous  complexities,  subtleties,  and  abstract  concepts; 
it  was  not  perfectly  obvious  that  truly  relevant  (or,  for  that 
matter,  irrelevant)  training  literature  would  be  recognized  as 
such .  Thus,  a  classification  scheme  for  both  training  issues  and 
training  literature  was  devised.  It  was  hoped  that  by  identify¬ 
ing  the  class  of  issues  of  which  a  TEC  CTEA  would  be  but  one 
example,  TEC-relevant  literature  could  be  identified  as  all  litera¬ 
ture  dealing  with  that  class  of  issues.  Based  on  a  "resources  - 
effectiveness  hierarchy",  a  cause-effect  model  linking  resources 
to  battalion  effectiveness  through  a  number  of  intermediate  steps, 
four  classes  of  training-relevant  issues  were  identified.  These 
classes,  developed  and  discussed  in  the  next  section,  were  then 
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used  to  categorize  the  training  literature  abstracts  (see  Appendix). 


The  results  of  the  literature  survey  were  mixed.  Methodology 
directly  and  specifically  applicable  to  the  evaluation  of  the  TEC 
program  was  not  uncovered  in  the  course  of  the  survey.  While  some 
of  the  concepts  and  techniques  which  surfaced  should  provide  general 
guidance,  and  while  others  should  prove  useful  as  part  of  an  even¬ 
tual  TEC  CTEA  methodology,  that  TEC-specific  methodology  apparently 
still  awaits  construction. 


Framework  for  Assessment  of  Training  Literature. 

Our  objective  in  this  section  is  to  develop  a  convenient 
framework,  or  set  of  logical  categories,  by  which  to  organize  the 
vast  literature  dealing  with  evaluation  of  training.  The  motiva¬ 
tion  for  this  taxonomy  is  to  better  determine  which  literature 
can  apply  to  the  evaluation  of  the  TEC  program,  by  noting  where  in 
the  taxonomy  TEC  fits;  and  then  noting  the  corresponding  literature 
Our  approach  is  to  first  construct  a  conceptual  hierarchy  relating 
resources  to  battalion  effectiveness,  then  to  categorize  training 
issues  with  reference  to  the  hierarchy;  and  finally  to  discuss  the 
relation  of  the  TEC  assessment  to  the  hierarchy. 


The  Resources-Ef feet i veness  Hierarchy. 

Figure  1  illustrates  the  Resources-Ef feet i veness  Hierarchy. 

It  indicates  (starting  at  the  base)  that  various  resources  are 
used  to  produce  various  training  programs.  The  set  braces,  {}, 
indicate  there  is  a  set  of  resources  and  a  set  of  training  programs. 
In  general,  each  training  program  is  produced  from  a  subset  of 
the  resources,  and  particular  resources  may  be  used  in  nore  than 
one  training  program.  The  next  step  up  the  hierarchy  suggests 
that  various  training  programs  produce  various  individual  task 
proficiencies.  The  process  continues  up  to  the  top  step  which 
suggests  that  battalion  proficiency  in  its  various  missions  con¬ 
tributes  to  a  single  ultimate  measure  of  battalion  effectiveness.5 


Classes  of  Issues. 


The  Resources-Ef feet ivuness  Hierarchy  is  a  convenient  way  to 
lend  organization  to  the  training  literature,  insofar  as  it  sug- 


5 

The  hierarchy  of  Figure  1  is  hardly  unique.  Truncated  or  ex¬ 
tended  versions  could  easily  be  constructed.  For  what  follows,  the 
specific  form  of  the  hierarchy  is  not  critical.  We  need  only  recog¬ 
nize  the  existence  of  some  such  hierarchy. 
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Battalion  Effectiveness 


Fiyure  1 


<- 

{  Battalion  Mission  Proficiencies  } 


{  Company  Task  Proficiencies  } 
{  Platoon  Task  Proficiencies  } 
{  Squad  Task  Proficiencies  } 
{  individual  Task  Proficiencies  } 
{  Training  Programs  } 
{  Resources  ] 


The  Rcsources-Ef feet i veness  Hierarchy 
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gests  several  (more  or  less)  non-overlapping  classes  of  training 
issues  with  which  the  literature  is  concerned.  These  classes  of 
training  issues  can  be  described  as  follows. 


Production  Function  Issues  (Class  I). 

The  Resources-Ef fectiveness  Hierarchy  actually  represents  a 
series  of  production  functions.  For  example,  letting  A*,  A2l... 
represent  the  various  types  of  resources;  Blt  B2,...  represent  the 
various  types  of  training  programs;  Cj,  C2,...  represent  the  various 
types  of  individual  skills;  etc.;  relationships  of  the  following 
forms  exist: 

-  i  •  i 

Bj  =  B  (A,,  A2,...)  V; 


C.  -  C'(B, 
etc . 


Y--> 


V. 


where  B.  is  the  "quantity"  of  B.  produced,  and  A.  is  the  quantity  of 

_  __  i 

Aj  used  to  produce  Bj,  and  likewise  for  C.and  B . .  Additionally, 

"nested"  production  functions  exist: 

11  2  2  2 
3  I 

2  1  2 


i/J 


C.  =  C  (B  (A.  A  ,...),  B  (A  ,  A  ,...) 
«  |  0  It 


which  relate  the  elements  of  non-adjoining  sets  in  the  hierarchy. 

A  great  deal  of  the  training  literature  is  concerned  with 
production  function  issues,  although  the  literature  is  rarely 
self-identified  with  that  appellation.  For  example,  literature 
dealing  with  questions  like: 

o  does  this  training  program  contribute  to  these 
skills  or  tasks? 

o  which  of  these  several  training  programs  contribute 
most  to  this  skill  or  this  task? 

o  which  resources  best  contribute  to  the  development 
of  cer ta in  skills? 

o  which  individual  task  proficiencies  contribute  to 
group  task  proficiencies? 
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is,  in  fact,  inquiring  as  to  the  arguments  and  form  of  one  of 
the  production  functions  suggested  by  Figure  J. 


Cost  Estimation  I ssues  (Class  II). 


Associated  with  each  element  in  each  set  in  the  hierarchy  are 
cost  estimation  problems,  such  as: 

o  what  is  the  expected  cost  of  a  specific  training 
program? 

o  what  has  been  the  cost  of  producing  specific  levels 
of  individual  skills? 

o  what  is  the  range  in  costs  of  achieving  a  given 
level  of  platoon  task  proficiency? 

Even  a  casual  survey  indicated  a  good  deal  of  literature  treats 
cost  estimation  issues. 


Efficiency  Issues  (Class  111). 

By  efficiency  is  meant  how  best  to  achieve  some  end.  There  are 
three  types  of  efficiency  issues: 

o  Maximize  Output  given  a  prescribed  level  of  costs  for 
the  inputs,  and  given  the  production  function  which 
determines  the  input/output  possibilities. 

o  Minimize  Costs  given  a  prescribed  level  of  output,  and 
given  the  production  function. 

o  Determine  the  Optimal  Output/Cost  combination  given 
the  minimum  cost  of  any  specified  level  of  output. 

Of  these  three  types  of  efficiency  issues,  the  first  two  are  as¬ 
sociated  with  the  rubric  cost-effectiveness  analysis  (CEA)  ,  while 
the  third  is  known  as  cost-benefit  analysis  (CBAj.  Cost-effec¬ 
tiveness  analyses,  insofar  as  they  deal  with  the  manipulation  of 
directly  observable  functions  (cost  and  production),  are  usually 
conceptually  straight-forward.  On  the  other  hand,  cost-benefit 
analyses  deal  with  the  preferences  (or  utility)  of  a  reference 
agent  or  organization.  These  preferences  are  largely  unobserved, 
and  must  be  somehow  inferred.  Needless  to  say,  cost-benefit  analy¬ 
ses  are  usually  difficult  to  carry  out. 

Much  of  the  training  literature  is  concerned  with  efficiency 
issues,  and  most  reports  are  of  the  cost-effectiveness  variety. 
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Valuation  Issues  (Class  IV). 


A  class  of  issues  related  to  the  preceding  class  is  the 
valuation  of  hierarchical  elements.  For  example,  one  night  ask 
what  the  value  of  a  specific  training  program  is,  or  what  the 
value  of  an  increment  in  some  task  proficiency  is.  The  link  be¬ 
tween  this  class  and  the  previous  one  is  that  both  directly  ad¬ 
dress  policy  questions;  both  deal  with  determining  what  should  be 
done  -  what  type  or  level  of  training  (or  skill  or  task  profi¬ 
ciency)  should  be  achieved?  Additionally,  both  deal  with  CEA 
and  CBA.  In  this  set  of  issues,  CEA  and  CBA  provide  the  methods 
to  arrive  at  valuations.  Two  subclasses  of  issues  may  be  iden¬ 
tified  here.  These  are  where  the  program  (or  skill  or  task)  to 
be  valued  is  essentially  a  substitute  for  ether  programs  which 
perform  the  same  function;  or  where  the  program  is  essentially 
a  complement  to  other  programs,  extending,  amplifying,  or  rein¬ 
forcing  some  skill  levels  or  proficiencies.  In  the  first  case, 
where  the  program  is  a  substitute  for  other  programs,  the  methods 
of  CEA  may  be  used  to  determine  value  by  computing  cost  savings 
or  increases  in  output.  In  the  second  case,  the  more  difficult 
procedures  of  CBA  must  he  employed  to  derive  a  value. 


The  Training  Extension  Course  (TEC)  Program. 

"TEC  is  a  program  that  has  been  designed  to  help  put  into  the 
hands  of  trainers,  both  in  units  and  in  institutions,  exportable, 
per formance-or iented  training  packages.  It  is  designed  to  pro¬ 
vide  soldiers  and  their  commanders  with  immediate  access  to  high 
quality,  self-paced,  multimedia  instruction  especially  designed 
to  assist  in  acquiring  and  maintaining  skills  critical  to  the 
soldier's  on-the-job  performance.  Each  TEC  lesson  is  individually 
packaged,  and  ready  for  use  on  an  individual  basis,  or  by  small 
groups,  under  the  supervision  of  an  NCO. 

"Historically,  TEC  began  in  the  early  1970’s  as  a  response 
to  a  number  of  problems  in  individual  training.  The  most  serious 
of  these  was  detected  by  the  U.  S.  Army  Board  for  Dynamic  Train¬ 
ing.  They  found  that  many  soldiers  in  combat  arms  units  were 
not  able  to  perform  critical  tasks  in  their  primary  Military 
Occupational  Specialty  (MOS) .  This  lack  of  competency  on  the  part 
of  individuals  can  quickly  mushroom  and  destroy  a  unit's  combat 
capability.  In  the  process  of  considering  solutions  to  this  pro¬ 
blem,  many  other  problems  were  uncovered  that  confront  commanders, 
training  managers,  and  trainers.  Some  of  these  problems  were: 

o  insufficient  numbers  of  qualified  liCOs  to  conduct 
the  required  training 

o  demands  on  the  commander's  time  which  interferred 
with  training  time 
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o  personnel  turbulence  such  as  frequent  turnover 
and  low  manning  levels 

o  rapidly  changing  doctrine  which  necessitates 
changes  in  training 

o  limited  training  resources  such  as  the  shortages 
of  time,  personnel,  and  space  to  provide  sufficient 
field-based  training. 

"All  of  these  problems  underscored  the  need  for  a  system  of 
exportable  training  support  that  would  make  performance  based, 
high  quality  instruction  available  to  field  units  in  an  easy  to 
use  format.  TEC  was  conceived  as  such  a  system"  (Litton  Systems, 
Inc.,  1978). 

The  issue  addressed  by  this  research  program  is  the  value  of 
the  TEC  program.  Th*  question  at  this  point  is  where,  in  the 
foregoing  issue  taxonomy,  does  this  research  program  fall?  Clearly, 
it  falls  within  our  Issue  Class  IV.  Moreover,  because  the  TEC 
program  is  not  designed  as  a  substitute  or  replacement  for  any 
other  s p<  c i f i c  training  program,  but  rather  is  meant  to  enhance 
training  m  field  units,  our  research  program  falls  into  the 
Complement  subclass  of  Issue  Class  IV.  This  is  not  to  suggest 
that  TEC  could  not,  will  not,  or  does  not  substitute  for  some 
amounts  of  some  other  types  of  training  in  field  units.  Vnat  is 
meant  is  that  insofar  as  at  least  some  TEC  usage  is  complementary, 
the  value-determining  methodology  must  be  capable  of  evaluating 
TEC  in  its  complementary  role.  This  means  that  the  methods  of 
cost-benefit  analysis,  rather  than  simply  cost-effectiveness 
analysis,  must  be  brought  to  bear  on  the  problem.  Additionally, 
this  means  the  evaluation  of  the  TEC  program  can  be  expected  to 
present  the  same  substantial  di ff icul ti tes  present  in  most  cost- 
benefit  studies. 

These  difficulties  can  be  illustrated  by  reference  to  a 
vertically  truncated  but  horizontally  expanded  version  of  the 
Resources-Ef fecti veness  Hierarchy,  such  as  in  Figure  2.  Sup¬ 
pose  that  individuals  are  currently  receiving  some  Baseline 
Training  Program  to  produce  proficiency  in  some  task,  such  as 
Task  2  in  the  figure.  Suppose  an  alternative  training  program 
is  proposed  as  a  substitute  for  the  baseline  program.  To  value 
that  proposed  alternative  program  we  need  only  determine  whether 
that  program  achieves  the  same  task  proficiency  at  less  cost. 

The  difference  in  cost  is  the  relative  net  value  of  the  alter¬ 
native  program.  The  information  requirements  to  carry  out  the 
analysis  are  limited  to  the  cost  functions  for  both  training 
programs  and  the  production  functions  linking  each  training  pro¬ 
gram  to  individual  task  proficiencies.  Such  information  re- 
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quirements  are  hardly  trivial,  yet  they  are  subs  tan t i a  1 1 y  less 
than  if  the  proposed  program  were  intended,  at  least  partially, 
to  complement  the  baseline  program.  Insofar  as  complementarity 
is  intended ,  then  greater  total  program  costs  along  with  grea ter 
task  proficiency  is  i ntended ,  and  it  is  meaningless  to  ask  whe¬ 
ther  the  proposed  program  achieves  the  same  task  proficiency  at 
less  cost.  What  must  be  asked  is  whether  the  greater  task  pro¬ 
ficiency  is  worth  the  greater  cost.  Note  that  the  implied  com¬ 
parison  between  greater  proficiency  and  greater  cost  is  between 
magnitudes  of  entirely  different  dimensions.  Note  also  that  in 
the  previous  case  where  the  proposed  program  is  an  al tern at i ve 
(i.  e.,  substitute),  the  resulting  comparison  is  between  two 
cost?  -  clearly  involving  the  same  dimension.  Thus,  the  critical 
difference  between  the  substitute  case  and  the  complement  case 
is  the  resulting  d imens ionai i ty  of  the  magnitudes  intended  for 
comparison.  This  is  largely  the  difference  between  CCA  and  C3A. 
The  latter  almost  inherently  deals  with  " i ncommensurabl es",  while 
the  former  may  not. 


An  Observation  About  Training  Program  Analysis. 

When  different  dimensions  arise  in  the  course  of  a  training 
program  analysis,  some  transformation  must  be  found  to  "map"  each 
dimension  into  one  common  dimension.  The  common  dimension  is 
typically  dollars  and  the  transformation  is  sometimes  known  as 
"shadow  pricing".  While  there  are  many  shadow  pricing  techniques, 
in  the  case  of  training  one  would  attempt  to  trace  the  effects  of 
the  greater  task  proficiency  to  some  other  set  of  effects  which  are 
amenable  to  dollar  valuation,  and  by  holding  all  other  factors  con¬ 
stant,  ascribe  that  dollar  value  back  to  the  training  program 
under  scrutiny. 

This  point  can  be  clarified  by  example.  Consider  a  ver¬ 
tically  integrated  firm  which  provides  some  of  its  own  inputs 
to  its  production  process  rather  than  purchasing  those  inputs  in 
the  marketplace.  Suppose  that  for  some  internal  accounting  or 
control  purposes  the  firm  wishes  to  place  a  meaningful  dollar 
value  (per  unit)  on  one  of  its  self-supplied  inputs.  The  most 
reasonable  approach  is  to  trace  the  impact  of  that  input  through 
the  production  process  and  on  the  value  of  the  final  gocd(s)  pro¬ 
duced.  The  value  of  the  input  is  its  contribution  to  the  value 
of  the  final  product.  In  the  parlance  of  traditional  economics, 
this  is  the  input's  marginal  revenue  product;  in  the  parlance  of 
mathematical  programming,  this  is  the  input's  dual  variable's 
value  (Sassone,  1977). 


Another  example,  somewhat  closer  in  spirit  to  the  issue  at 
hand,  would  be  the  attempt  to  place  a  value  on  a  computer  based 


instructional  system  used  in  a  government  vocational  training  pro¬ 
gram  for  low  income  civilians.  If  the  object  of  the  program  is  to 
increase  the  trainees'  incomes,  the  value  of  the  computer  based 
instruction  system  would  be  its  marginal  contribution  to  the  sum 
of  the  present  value  of  the  trainees'  incomes. 

In  each  of  these  examples,  a  single  production  function  is  in¬ 
volved,  and  the  output  of  the  production  process  is  readily  expres¬ 
sible  in  monetary  terms.  In  the  first  example  the  production  func- 
is  the  relation  between  the  firm's  inputs  and  its  final  output,  in 
the  second  example  the  production  function  is  the  relation  between 
the  resources  used  for  training  and  the  trainees'  incomes.  It  is 
clear  that  in  both  cases  outputs  are  easily  expressed  in  dollar 
terms.  The  problems  associated  with  imputing  values  to  the  inputs 
in  these  cases  would  revolve  around  the  determination  of  the  form 
of  the  specific  production  function  involved  -  no  mean  task.  Yet, 
these  two  cases  are  far  simpler  than  the  TEC  case  because: 

o  TEC  evaluation  involves  all  the  production  functions 
depicted  in  the  Resources  -  Effectiveness  Hierarchy 
of  Figure  1,  not  just  one 

o  the  output  of  none  of  the  production  processes  of 
Figure  1  is  readily  expressible  in  monetary  terms 

These  complications  mean  that  the  effects  of  TEC  must  be  traced 
through  a  number  of  production  functions. 

The  more  production  functions  become  involved,  the  more  varia¬ 
bles  must  be  standardized,  the  more  tenuous  become  the  relations, 
and  the  more  errors  are  likely  to  find  their  way  into  the  analysis. 
And  at  some  point,  an  "effectiveness  to  dollars"  transformation 
must  be  carried  out.  In  sum,  the  requirements  for  the  evaluation 
of  a  complementary  training  program,  such  as  TEC,  are  very  substan¬ 
tial;  and  the  analysis  is  likely  to  be  very  difficult  to  carry  out. 


Discussion  of  Training  Literature. 


A  review  of  both  the  military  and  the  civilian  training  edu¬ 
cation  literatures  failed  to  uncover  a  training  evaluation  method 
oloyy  directly  applicable  to  TEC.  This  finding  should  not  be  con 
strued  os  an  indictment  of  the  literature,  rather  it  is  a  reflec¬ 
tion  of  TEC's  unique  characteristic  -  its  intended  comol ementar i t 
to  existing  training  systems. 


A  convenient  way  to  place  the  training  literature  in  perspec 
tive  is  to  compare  it  with  the  TEC  CTEA  requirements  identified 
above,  and  determine  to  what  extent  such  requirements  are  addres¬ 
sed  within  the  literature.  While  five  TEC  CTEA  requirements  are 


mentioned,  the  first  t wo  (identification  of  the  baseline  and  TEC- 
modified  training  systems)  are  problem  specific,  and  cannot  rea¬ 
sonably  be  expected  to  have  been  previously  addressed.  The  third 
requirement  is  an  input  cost  function,  relating  training  activities 
(such  as  garrison  training  days,  and  specific  types  of  field  train¬ 
ing  days)  to  the  total  costs  of  carrying  out  those  activities. 

While  a  number  of  documents  reviewed  nominally  dealt  with  training 
cost  estimation,  their  focus  was  on  either  very  general  procedures 
or  actual  cost  estimation  for  specific  hardware  items.  They  ap¬ 
peared  not  to  deal  with  the  aggregate  costs  of  the  complete  train¬ 
ing  system.  For  example,  Kopstein  and  Seidel  (1967),  deals  with 
estimating  the  costs  of  computer  instruction,  and  both  Walker  (197*0 
and  Shriver  and  Hart  (1975)  deal  with  general  models  of  life  cycle 
costing.6 

The  fourth  identified  TEC-CTEA  requirement  is  an  "effectiveness 
production  function"  which  can  relate  any  given  levels  of  any  types 
of  training,  along  with  other  relevant  variables,  to  a  unit's  (per¬ 
son's,  platoon's,  battalion's,  etc.)  combat  effectiveness.  This 
need  for  a  production  function  relationship  between  training  and 
effectiveness  has  widespread  recognition  (Braby,  et.  al.,  1§72; 

Braby,  et.  al.,  1975;  USATRADOC,  1975;  Lukasczyk,  1976),  yet  ap¬ 
parently  due  to  conceptual  and  practical  difficulties,  such  a  pro¬ 
duction  function  has  yet  to  be  constructed.  That  literature  which 
has  been  identified  herein  as  being  associated  with  production  func¬ 
tions  (see  abstracts)  generally  deals  with  the  comparative  charac¬ 
teristics  of  different  production  functions,  but  not  with  the  esti¬ 
mation  of  effectiveness  production  functions.  For  example,  the 
Proceedings  of  NTEC/ I ndus try  Conference  (NTEC,  1975),  contains  several 
studies  wherein  the  relative  impacts  of  different  types  of  training 
on  individual  effectiveness  were  examined.  Such  studies  can  be 
interpreted  as  casting  light  on  the  nature  of  certain  partial 
derivatives  of  production  functions,  but  do  not  address  the  nature 
of  the  whole  function,  nor  the  impacts  on  group  effectiveness.  A 
production  function  is  estimated  in  Weiher  and  Horowitz,  (1971),  but 
this  is  not  an  effectiveness  production  function.  Rather,  it  is  a 
relation  between  the  training  inputs  employed  and  the  number  of 
trained  recruits  produced. 

The  fifth  requirement  for  3  TEC  CTEA  is  a  transformation  which 
can  translate  changes  in  effectiveness  into  corresponding  dollar 
values.  Such  a  transformation  appears  to  be  absent  from  the  train¬ 
ing  literature.  This  absence  has  a  simple  explanation,  however. 


6 

TEC-relevant  cost  estimating  literature  may  be  available  in  the 
future  as  a  result  of  current  research  in  the  ODCSCOMP  FORSCOM  at 
Fort  McPherson,  Georgia. 
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Most  new  training  programs  which  have  undergone  the  scrutiny  of 
an  economic  analysis  (see  abstracts  dealing  with  efficiency  and 
valuation  issues)  have  been  by  nature,  or  have  been  analyzed  as, 
substitute  training  programs.  That  is,  the  choice  was  between 
that  program  and  some  other.  Both  could  not  be  chosen.  In  that 
context,  it  is  meaningful  to  ask,  if  dollar  resources  are  fixed, 
which  program  yields  more  effectiveness  for  the  money;  or  if  the 
effectiveness  level  is  fixed,  which  program  costs  less  to  achieve 
that  effectiveness  level.  Braby,  et.  al.,  1972;  Braby,  et.  al .  , 

1975;  Lukasczyk,  1976;  and  Toomepuu,  1977;  are  all  studios  which 
are  examples  of  dealing  with  training  programs  as  substitutes. 

In  sum,  because  of  TEC's  unique  complementary  role,  it  does 
not  fit  the  usual  training  program  mold,  and  analytical  techniques 
currently  available  in  the  training  literature  are  not  completely 
suitable  for  a  TEC  evaluation.  To  be  sure,  general  guidance  for 
a  TEC  CTEA  is  available  from  documents  such  as  the  TRASANA  CTEA 
Handbook,  USATRADOC,  1975,  and  Braby,  et.  al.,  1975.  Nonetheless, 
it  appears  a  TEC-specific  evaluation  methodology  must  be  developed. 

ECONOMIC  LITERATURE 

In  contrast  to  the  framework  developed  for  training  literature 
no  such  framework  was  deemed  necessary  for  the  economic  evaluation 
1 i terature. 

This  literature  makes  two  general  contributions  to  the  develop¬ 
ment  of  a  methodology  for  TEC  evaluation.  First,  it  provides  a 
rational  structure  within  which  TEC  evaluation  can  be  carried  out. 
While  each  author  might  emphasize  somewhat  different  aspects  of 
this  structure,  a  single  consistent  and  useful  point  of  view,  nonethe¬ 
less,  emerges.  TEC  should  be  evaluated  using  the  so-called  "with/ 
without"  approach7  (Sassone  and  Shaffer,  1978;  McKean,  1958;  Das- 
gupta  and  Pearce,  1972;  Eckstein,  1958;  Hirshliefer,  et.  al.,  I960; 
Krutilla  and  Eckstein,  1958;  Little  arid  Mirrlees,  197M  which  in¬ 
volves  the  following  steps: 

o  Definition  of  Problem 

This  involves  careful  specification  of  the  ob¬ 
jectives,  constraints,  and  the  relevant  soci  ety 
for  which  costs  and  benefits  accrue. 


7 

"With/without"  means  comparing  the  state  of  affairs  with  the 
proposed  project  against  the  state  of  affairs  wi thout  the  project. 
This  is  in  contrast  to  a  "before/after"  approach,  for  example. 


o  Identification  of  Alternative  and  Baseline  Programs 

This  is  the  specification  of  the  programs  or 
projects  against  which  the  proposed  project  is 
to  be  compared.  As  a  minimum,  the  "status  quo" 
is  usually  considered  as  the  baseline  for  com¬ 
parison 

o  Construction  of  Project  Scenario 

A  complete,  quantitative  and  qualitative,  and 
chronological  description  of  resource  inputs 
and  outputs,  and  other  economic/social  ramifi¬ 
cations  of  the  proposed  project,  must  be  con¬ 
structed 

o  Construction  of  Alternative  (or  Baseline)  Scenarios 

A  complete,  quantitative  and  qualitative,  and 
chronological  description  of  the  Alternative 
(or  Baseline)  Scenario  must  be  constructed 

o  Identification  of  Differences  Between  Scenarios 

The  differences  between  the  Project  and  Alter¬ 
native  (or  Baseline)  scenarios  must  be  recognized 
and  physically  quantified  to  the  extent  reasonable 
and  possible 

o  Valuation  of  Differences 

Using  market  prices  "shadow  prices"  (Dasgupta  and 
Pearce,  1972;  Sassone  and  Shaffer,  1978;  Mishan, 
1976;  Sassone,  1977).  the  physical  differences  be¬ 
tween  scenarios  should  be  valued  in  dollar  terns 
to  the  extent  reasonable  and  possible 

o  Aggregation  over  Time 

Using  an  appropriate  discounting  scheme,  (Sassone 
and  Shaffer,  1978;  Mishan,  1976;  Fcldscein,  1964; 
Marglin,  1963  a;  Marglin,  1963  b;  Eaumol ,  1 9 6 8 )  , 
the  valuations  associated  with  different  tine 
periods  must  be  aggregated  into  a  sirgle  "figure 
of  mer i t" 

o  Sensitivity  Analysis 

The  sensitivity  of  the  derived  figure  of  merit  to 
various  assumptions  and  estimates  should  be  examined 
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o  Final  Report 


I 


I 


I 


I 


The  information  generated  by  the  analysis  should 
be  presented  to  relevant  decision  makers  in  an 
accessible  and  digestable  format 

By  and  large,  the  military  training  literature  which  addresses  the 
structuring  of  a  project  evaluation  is  consistent  with  the  spirit 
of  the  foregoing  steps,  although  -  as  should  be  expected  -  the 
specific  steps  mentioned  in  that  literature  are  tailored  to  speci¬ 
fic  mi  1 i tary  issues . 

The  second,  and  more  important,  contribution  to  the  development 
of  TEC  evalution  methodology  made  by  the  economics  and  cost-benefit 
literature  is  the  suggestion  of  a  way  to  place  meaningful  dollar 
values  on  changes  in  effectiveness  wrought  by  changes  in  training. 
The  approach  involves  the  calculation  of  compensated  and  equivalent 
budget  variations.  Each  author  presents  these  concepts  somewhat 
differently,  and  often  cryptically  (Currie,  et.  al.,  19&5;  Sanuel- 
son,  1965;  Mishan,  1976;  Henderson  19^1;  Hicks,  1944;  Dupuit,  1968; 
Hicks,  1956).  The  full  f’^vor  of  these  useful  but  highly  technical 
concepts  are  conveyed  in  the  development  of  the  training  evaluation 
model  to  be  presented  in  a  subsequent  report. 


SUMMARY  AND  CONCLUSIONS 

This  report  is  the  first  of  a  set  of  two  reports  dealing  with 
CTEA  methodology  directed  at  the  evaluation  of  the  TEC  program.  This 
Jocument  reports  on  a  review  and  evaluation  of  literature  relevant 
to  a  TEC  CTEA.  The  objective  of  the  review  is  to  identify  State- 
of-the-Art  methods  and  techniques  applicable,  in  whole  or  part,  to 
the  performance  of  a  TEC  CTEA.  The  second  document  in  this  set  des¬ 
cribes  the  methodology  formulated,  in  light  of  this  review,  to  carry 
through  a  TEC  CTEA.  This  present  document  can  be  viewed  as  back¬ 
ground  and  introduction  to  that  second  document. 

The  TEC  Program  was  designed  as  a  series  of  multimedia,  ex¬ 
portable,  performance-oriented  training  packages.  Its  purpose  is 
to  provide  high  quality,  uniform  instruction  in  critical  skills  to 
individual  and  small  groups  of  soldiers.  A  TEC  CTEA  must  involve, 
like  any  CTEA,  a  comparison  of  the  effectiveness  and  costs  of  al¬ 
ternative  training  systems.  A  CTEA  must  relate  a  (proposed  or  ac¬ 
tual)  change  in  the  training  system  to  its  ultimate  impact  of  com¬ 
bat  effectiveness.  There  should  be  a  clear  "audit  trail"  from  the 
former  to  the  latter.  In  the  case  of  TEC,  a  complementary  trainii  g 
system  focused  on  the  individual  soldier,  establishing  such  an  audit 
trail  is  likely  to  be  a  difficult  undertaking.  It  involves  tracing 
the  impact  of  TEC  through  a  number  of  "production  function"  relation¬ 
ships  to  some  final  impact  on  unit  (not  individual)  combat  effective- 
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ness.  In  a  CTEA,  the  proposed  alternative  must  be  compared  with 
the  baseline  system.  The  comparison  can  be  of  three  different 
types:  fixed  cost,  variable  effectiveness;  variable  costs,  fixed 
effectiveness;  or  variable  costs,  variable  effectiveness.  Because 
TEC  is  a  compl cmentary  training  system,  its  evaluation  must  be  of 
the  last  variety.  That  is,  because  TEC  is  designed  to  be  used 
with  existing  training  systems,  and  not  necessar i 1 y  to  replace  any 
baseline  training,  TEC  is  intended  to  increase  both  costs  and  ef¬ 
fectiveness.  Its  CTEA  must,  of  course,  reflect  its  designated  ob¬ 
jective  and  must  determine  whether  the  increase  in  effectiveness 
is  worth  the  increase  in  costs. 

Five  methodological  requirements  for  a  TEC  CTEA  can  be  speci¬ 
fied.  These  are: 

o  the  baseline  training  system  must  be  identified  and 
its  parameters  quantified 

o  the  training  system  inclusive  of  TEC  must  be  defined 
and  its  parameters  quantified 

o  an  input  cost  function  must  be  developed  which  maps 
any  training  system  configuration  into  its  dollar 
cost 

o  an  effectiveness  production  function  must  be  developed 
which  maps  any  training  system  configuration  into  an 
objective  measure  of  its  effectiveness 

o  an  approach  must  be  devised  which  objectively  trans¬ 
lates  changes  in  effectiveness  into  corresponding 
dol 1 ar  va 1 ues . 

In  order  to  determine  whether  any  or  all  of  these  TEC  CTEA 
methodological  requirements  could  be  satisfied  from  existing 
literature,  a  literature  search  was  conducted.  Two  more  or  less 
separate  bodies  of  literature  were  searched  for  TEC  CTEA  -  rele¬ 
vant  material.  The  first  body  was  the  training  evaluation  litera¬ 
ture.  This  literature,  comprised  primarily  of  military  documents, 
is  problem  oriented.  It  generally  focuses  on  a  specific  training 
program.  In  contrast,  the  second  body  of  1 i terature  was  primarily 
theoretical  in  orientation.  This  was  the  economic  evaluation 
literature,  comprised  of  monographs  and  journal  articles  dealing 
with  the  general  economic  considerations  of  evaluating  any  type 
of  pmgram  or  project.  Because  of  inherent  differences  between 
these  two  bodies  of  literature,  different  search  procedures  were 
used  in  each  case  to  ferret  out  TEC  -  relevant  material.  For  the 
training  literature,  computerized  literature  searches  were  employed 
to  generate  extensive  lists  of  titles  and  abstracts.  These  were 
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screened  on  the  basis  of  apparent  TEC  relevance,  and  those  sur¬ 
viving  the  screening  were  reviewed  in  depth.  The  economic 
evaluation  literature  was  selected  from  the  subject  classification 
indices  of  the  Journal  of  Economics  Literature,  from  the  biblio¬ 
graphies  of  recent  comprehensive  monographs  in  project  evaluation, 
and  from  among  the  well-known  "classics"  of  the  field. 

A  convenient  framework  providing  a  conceptual  organization 
of  the  training  literature  was  developed.  It  divides  the  train¬ 
ing  literature  into  four  more  or  less  mutually  exclusive  cate¬ 
gories.  These  categories  include  literature  dealing  with  the 
nature  of  effectiveness  production  functions  (Class  1),  cost 
estimation  issues  (Class  II),  efficiency  issues  (Class  III),  and 
valuation  issues  (Class  IV). 

Methodology  directly  applicable  to  the  evaluation  of  the  TEC 
Program  was  not  uncovered  in  the  course  of  the  survey  of  the  train¬ 
ing  literature.  However,  a  number  of  concepts  and  techniques  which 
surfaced  should  prove  useful  as  "building  blocks"  in  the  eventual 
constructing  of  a  TEC  CTEA  methodology. 

Training  literature  relevant  to  TEC  Program  evaluation  can 
be  expected  to  fall  into  Classes  III  and  IV,  primarily  the  latter. 
With  regard  to  the  Class  111  literature,  it  deals  almost  exclusively 
with  Cost-Effectiveness  issues.  And  as  was  pointed  out,  a  TEC 
evaluation  must  be  attacked  on  a  Cost-Benefit  basis.  There  are 
two  reasons  why  the  Class  IV  literature  is  not  germane.  One  reason 
is  that  the  literature  deals  with  the  "Substitute"  subclass  of  Class 
IV,  whereas  TEC  falls  into  the  "Complement"  subclass.  Indeed,  the 
sole  military  document  in  our  Class  IV  deals  with  the  "Substitute" 
ca^e.  However,  the  primary  reason  is  that  even  the  "Complement" 
literature  deals  with  the  evaluation  of  training  programs  in  terms 
of  the  impict  of  the  program  on  individuals'  future  earning  poten¬ 
tial.  This  is  entirely  appropriate  for  civilian  training  programs 
where  the  objective  is  either  increased  production8  or  increased 
income  for  the  worker.  However,  insofar  as  military  training  pro¬ 
grams  -  and  the  TEC  Program  in  particular  -  have  the  objective  of 
increasing  combat  effectiveness,  and  insofar  as  a  soldier's  mili- 


e 

In  the  private  sector  it  car.  generally  be  assumed  that  an  in¬ 
dividual's  wage  is  a  measure  of  his  productivity.  Therefore,  when 
an  individual's  marginal  contribution  to  production  cannot  be  direc¬ 
tly  estimated,  his  market  determined  wage  is  a  useful  surrogate. 
Thus,  the  effects  of  training  on  production  can  be  inferred  by 
studying  changes  in  market  wage  rates. 
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tary  wage  cannot  be  presumed  to  measure  his  marginal  contribution 
to  combat  effectiveness,  the  civilian  literature  would  appear  to 
have  little  bearing  on  TEC  evaluation. 

The  major  contribution  of  the  training  literature  to  a  TEC 
evaluation  is  in  the  prescription  of  a  general,  rather  than  spe¬ 
cific,  course  of  analysis.  Such  documents  as  the  TRASA1JA  CTEA 
Handbook,  USATRADOC,  1975,  and  Braby,  et.  al.,  1975  present  solid 
guidance  regarding  the  philosophy,  orientation,  and  direction  of 
a  TEC  CTEA. 

The  economic  evaluation  literature  makes  two  general  contri¬ 
butions  to  the  development  of  a  methodology  for  TEC  evaluation. 
First,  it  provides  a  rational  structure  within  which  a  TEC  evalu¬ 
ation  can  be  carried  out.  This  is  the  structure  of  cost-  benefit 
analysis  which  a  large  number  of  authors  have  contributed  to  over 
the  past  several  decades.  Second,  this  literature  suggests  a  way 
to  place  meaningful  dollar  values  on  changes  in  effectiveness 
brought  about  by  changes  in  the  training  system.  This  approach 
involves  the  estimation  of  an  effectiveness  production  function, 
and  the  calculation  of  compensating  and/or  equivalent  budget 
var i at  ions . 

The  contributions  of  both  the  training  and  economic  evaluation 
literatures  to  a  TEC  CTEA  are  general  rather  than  specific.  Both 
literatures  provide  useful  outlines  for  constructing  a  TEC  CTEA 
methodology,  and  the  economic  evaluation  literature  provides  a 
means  of  translating  effectiveness  into  a  meaningful  dollar  equi¬ 
valent.  What  remains  to  be  developed  are  the  al 1 -impor tant  de¬ 
tails.  Spec i f i ca 1 ly : 

o  data  availability  for  the  estimation  of  the  effective¬ 
ness  production  function  must  be  established 

o  efficient  means  of  data  collection  must  be  devised 

o  methods  of  data  reduc t ion/t rans format  ion  to  inter¬ 
face  the  data  and  the  methodology  must  be  developed 

o  the  best  methods  of  statistical  estimation  under 
various  conditions  of  data  availability  and  quality 
must  be  ident i f ied 

o  training  cost  estimation  procedures  must  be  developed 

o  mathematical  methods  for  the  derivation  of  the  com¬ 
pensated  and  equivalent  budget  variations  for  various 
functional  specifications  of  the  production  function 
mus  t  be  i den  t i f i ed 
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o  reasonable  methods  of  extrapolating  the  specific 
numerical  results  from  TEC  evaluation  lati tudinal ly 
(to  MOSs  other  than  those  specifically  dealt  with) 
and  longitudinally  (to  future  years)  must  be  developed 

o  the  best  interpretations  of  alternative  numerical 
results  must  be  worked  out 

o  contingency  methods  to  deal  with  data  unavai labi 1  i  ty 
or  other  problems  must  be  devised 

These  are  some  of  the  issue.;  an  ultimate  TEC  CTEA  methodology 
must  address. 
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ABSTRACTS 


This  appendix  provides  synopses  of  recent  literature  rele¬ 
vant  to  TEC  evaluation.  The  synopses  are  presented  in  two  groups: 
one  of  Training  Literature  and  one  of  Economic  Literature. 

TRAINING  LITERATURE 

Computerized  literature  searches  were  employed  to  identify 
that  training  literature  relevant  to  the  development  of  a  TEC 
CTEA  methodology.  Two  searches  of  the  National  Technical  Infor¬ 
mation  Service  (NT IS)  data  base  were  conducted,  which  resulted 
in  an  extensive  list  of  titles  and  abstracts  all  with  at  least 
some  tangential  relevant  training  evaluation.  The  abstracts  were 
reviewed,  and  a  set  of  approximately  1*0  documents  apparently  rele¬ 
vant  to  a  TEC  CTEA  was  identified.  These  documents  were  reviewed 
in  detail  and,  on  a  basis  of  actual  TEC  CTEA  relevance,  a  subset 
of  that  set  are  abstracted  herein. 

The  training  literature  is  arranged  according  to  the  Issue 
Classification  Framework  developed  in  the  text  (pp.  7“ll).  Need¬ 
less  say,  few  reports,  articles  or  books  fit  perfectly  into 
a  single  class.  Of  necessity,  a  certain  amount  of  judgement  must 
be  exercised  in  the  classification  process.  Where  a  piece  of 
literature  deals  in  a  substantial  way  with  more  than  one  issue, 
it  is  classified  according  to  its  primary  thrust,  but  its  treat¬ 
ment  of  other  issues  is  noted  by  brackets  after  the  title.  Within 
the  brackets  are  noted  the  other  classes  in  which  that  piece  of 
literature  might  reasonably  be  catalogued. 

The  reader  will  doubtless  note  that  much  of  the  training 
literature  surveyed  here  falls  into  our  Class  III.  This  is  no 
accident.  While  our  primary  interest  is  in  the  class  in  which 
TEC  evaluation  would  fall,  namely  Class  IV,  that  class  happens 
to  be  extremely  sparsely  populated.  Literature  most  closely  re¬ 
lated  to  Class  IV  is  in  Class  III,  and  that  set,  being  reasonably 
abundant,  is  therefore  accorded  a  substantial  representation. 

Literature  Dealing  with  Production  Function  Issues.  Weiher,  R. 
and  Horowitz,  S.  A.  A  Production  Function  for  Trained  Recruits. 
Center  for  Naval  Analyses.  Arlington,  Virginia:  November,  1971. 

The  purpose  of  this  study  was  to  determine  the  output  capability, 
or  capacity,  of  the  Navy's  initial  entry  training  bases  under  (then) 
current  and  alternative  operating  policies,  as  well  as  under  various 
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output  requirements  associated  with  alternative  force  levels. 

This  is  accomplished  by  estimating:  l)  the  feasible  output 
of  trainees  that  can  be  obtained  - » * . •  the  resources  currently 
at  the  recruit  training  bases;  2,  the  "surge"  capability  of  the 
bases  if  extra  instructors  are  assigned  to  the  recruit  training 
commands;  and  3)  the  capacity  of  the  recruit  training  establish¬ 
ment  for  any  combination  of  facilities  and  instructors  at  the 
bases.  The  paper  also  examines  the  question  of  scale  economics 
in  recruit  training.  The  analysis  centers  around  a  production 
function  where  the  number  of  trained  recruits  is  assumed  to  be 
a  function  of  the  value  of  the  physical  training  facilities,  the 
input  labor  (instructor  time),  and  the  number  of  input  recruits. 
Using  annual  data  for  the  years  1 964-1 969 ,  the  production  cunction 
was  estimated  using  a  least  squares  technique.  Among  the  conclu¬ 
sions  are  that  Navy  training  exhibits  constant  returns  to  scale, 
that  excess  recruit  training  capacity  existed  in  1970.  and  that 
savings  might  be  induced  if  recruit  loads  were  distributed  dif¬ 
ferently  among  the  training  bases. 


Proceedings  of  the  8th  UTEC/ I ndustry  Conference:  New  Concepts 
for  TTaining  Systems  (18-20  November  1 975) ,  Naval  Training  Equip- 
ment  Center  Technical  Report  1H-275  (Unclassified),  November  1975, 
(AO  A028  885).  The  conference  was  intended  to  provide  the  environ¬ 
ment  for  the  exchange  of  ideas  and  concepts  in  training  systems. 
Several  papers  are  mentioned  here. 

•  Evaluation  of  an  Automated  Flight  Training  System  (Puig,  J.  A. 
and  Gill,  S . )  .  This  paper  investigated  whether  training  with  the 
automated  GCA  module  was  as  effective  as  conventional  training.  An 
experimental  group  and  a  control  group  were  used.  In  each  case, 
training  was  to  proficiency  and  the  automatic  performance  measure¬ 
ment  capabilities  of  GCA  were  used,  in  addition  to  the  instructor 
scoring.  A  questionnaire  concerning  the  effectiveness  of  training 
with  the  GCA  module  (involving  instructors  and  students)  was  another 
evaluation  device.  The  results  indicated  that  training  with  the 
GCA  module  was  as  effective,  and  perhaps  more  effective,  than  con¬ 
ventional  training. 

•  USAF  Evaluation  of  an  Automated  Adaptive  Flight  Training  System 
( B  rown ,  J.  £.,  et.  al  . )  ,  The  major  objectives  of  the  reported  re- 
search  were: 

o  to  evaluate  the  training  effectiveness  of  the 
Automated  Flight  Training  System  (AFTS)  in  the 
F-4  Training  Program 

o  to  identify  desired  hardware  and  software  modifi¬ 
cations  for  operational  devices 

o  to  identify  effective  methods  of  operational  train¬ 
ing  use. 


One  of  the  major  characteristics  of  the  AFTS  was  its  use  of 
adaptive  training.  An  experimental  and  control  group  were  used. 
Performance  under  the  experienced  GCA  controllers  was  assumed 
to  represent  the  major  criterion  for  evaluating  the  training  ef¬ 
fectiveness  of  the  AFTS  system.  The  results  of  the  evaluation 
indicated  the  AFTS  to  be  an  effective  system  for  training  GCA's. 
The  cost-effectiveness  of  the  system  was  not  determined,  hov/ever 

•  An  Evaluation  of  Computer  Based  Instruction  for  Performance 
of  “Hands-On"  Training  Evolutions  (Radsken,  J.  W.  and  Grosson, 

X  F . )  .  This  paper  describes  an  evaluation  conducted  at  the 
Guided  Missile  School,  Dam  Neck,  Virginia  where  three  different 
computer  based  systems  were  used  to  compare  computer  based  in¬ 
struction  (C B 1 )  with  conventional  instruction.  The  objective 
was  to  ascertain  if  certain  "hands-on"  training  evolutions 
could  be  performed  on  computer  systems  and  thus  reduce  require¬ 
ments  for  tactical  training  equipment.  A  test  group  and  a  con¬ 
trol  group  were  used.  The  parameters  of  training  effectiveness 
were  time  to  train  and  level  of  knowledge  and  skill  capability. 
Cost  aspects  were  not  discussed  in  the  paper.  It  was  concluded 
that  CBI  is  a  feasible  approach  for  reducing  tactical  hardware 
utilization  for  interactive  operator  training  and  basic  main¬ 
tenance  training  tasks.  When  compared  to  conventional  instruc¬ 
tion,  CBI  provides  comparable  capability  in  knowledge  and  skill 
as  well  as  in  time  to  train. 


Demonstration  and  Evaluation  of  the  Plato  IV  Computer-Based 


Education  System;  Advanced  Research  Pro 


No.  2245  (Unclassified),  December  1975.  (AD  A024  b24)  [Secondary 
Class i f i cat  ion :  III].  This  semi-annual  report  describes  a  pro¬ 
gram  aimed  at  demonstration,  testing,  and  evaluation  of  the  edu¬ 
cational  effectiveness  of  Plato  as  implemented  in  several  geo¬ 
graphically  dispersed  military  training  sites.  It  also  describes 
program  of  technical  modifications  aimed  at  increasing  the  effec¬ 
tiveness  of  the  Plato  system.  It  summarizes  the  highlights  and 
major  events  of  the  ARPA/PLATO  project  for  the  second  half  of 

1975. 

Strasel ,  H.  C.,  ct.  al.  Training  Extension  Courses  (TEC) 
Costs  and  Training  Effectiveness,  U.  S.  Army  Research  Institute 
for  the  Behavioral  and  Social  Sciences,  Research  Report  Craft 
(Unclassified),  November  1977  [Secondary  Classification:  III]. 

The  report  summarizes  a  series  of  research  efforts  on  the  Traininr 
Extension  Course  (TEC)  program  conducted  under  the  sponsorship  of 
the  U.  S.  Army  Training  Support  Center  -  Training  Programs  Direc¬ 
torate.  This  program  included  detailed  research  into  the  train¬ 
ing  effectiveness  of  TEC,  the  current  and  programmed  costs  of  the 
TEC  program,  current  and  projected  usage  of  TEC  in  the  Active  and 


Reserve  components,  and  analytic  examination  of  the  cost  effec¬ 
tiveness  of  TEC.  The  analyses  and  results  are  presented  in 
relation  to  a  series  of  seven  questions  about  TEC  posed  by  Depart¬ 
ment  of  the  Army  Staff.  The  research  used  several  approaches  to 
answer  these  questions,  including  both  analytical  and  empirical 
approaches.  The  empirical  approaches  included: 

o  an  experimental  study  of  the  effectiveness  of  TEC 
training  and  the  retention  of  learning  with  TEC  in 
comparison  to  conventional  training  and  no  training, 
for  both  Active  and  Reserve  Components 

o  a  survey  of  EEC  usage  and  the  attitudes  and  problems 
expressed  by  TEC  users,  non-users,  training  officers 
and  NCOs,  and  Training  Support  personnel,  in  CONUS 
and  USAREUR  Active  units  and  Reserve  units 

o  an  analysis  of  the  relationship  of  TEC  lesson  usage 
and  SQT  performance  in  six  battalions  in  CONUS 

o  collection  of  all  identifiable  costs  related  to  pre¬ 
vious  TEC  program  and  projected  costs  on  a  uniform 
annual  cost  basis  for  the  next  ten  years. 

o  collection  from  TRADOC  Schools  of  ways  TEC  has  been 
used  to  supplement  or  substitute  for  conventional 
training  and  the  cost  savings  therefrom 

Using  these  and  other  data,  a  series  of  analyses  were  performed  to 
determine  the  potential  and  real  cost  effectiveness  of  TEC  train¬ 
ing  for  the  U.  S.  Army. 

The  report  concludes  that  TEC  is  both  cost  and  training  ef¬ 
fective  at  the  current  levels  of  training  effectiveness  and  the 
current  level  of  usage.  Analyses  further  indicate  that,  if  TEC 
usage  is  raised  through  command  emphasis  and  development  of  better 
methods  and  strategies  for  TEC  implementation,  the  cost  and  train¬ 
ing  effectiveness  of  TEC  will  be  greatly  increased.  Current  re¬ 
search  and  development  efforts  to  speed  the  better  implementation 
of  TEC  must  be  continued  to  realize  these  increases. 


Literature  Pea  line)  with  Cost  Estimation  Issues.  Ko[  s  t  e  i  n  ,  F .  F . 
and  Seidel,  R.  J.  Computer-Administered  Instruction  Versus  Tradi¬ 
tionally  Administered  Instruction:  Economics,  George  Washington 
University's  Human  Resource*  Research  Office,  Professional  Paper 
31-67  (Unclassified),  June  1^67,  (AD  656  613).  A  cost  compar i son 
of  computer-administered  instruction  (CAl)  versus  traditionally- 
administered  instruction  (TAl)  is  made  in  this  report  based  on  an 
assumption  of  equal  effectiveness  for  TAl  and  CAl,  The  mean  cost 
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per  student  hour  is  used  as  a  base  line.  TAI  costs  are  divided 
into  three  categories:  public  elementary  and  secondary  education, 
higher  education,  and  military  training.  Cost  figures  are  ob¬ 
tained  for  each  of  these  categories.  CAI  categories  are  hardware, 
maintenance  and  spare  parts,  software,  and  instructional  design 
and  program  preparation.  CAI  costs  per  student  per  hour  are  the 
total  costs  in  these  categories  during  operating  life  distributed 
over  the  total  number  of  students  who  make  use  of  the  system  dur¬ 
ing  each  one  of  its  life  hours.  Costs  are  obtained  for  each  cate¬ 
gory.  An  economic  comparison  is  then  made  of  CAI  and  TAI  costs. 

The  report  concludes  that  CAI  is  more  expensive  than  TAI  in  public 
education,  but  is  more  competitive  in  higher  education  and  mili¬ 
tary  training.  Among  the  educational  (in  contrast  to  economic) 
considerations  discussed  are  that  CAI  can  potentially  adapt  the 
instructional  presentation  not  only  to  the  detailed  characteristics 
of  the  trainee,  but  also  to  his  precise  requirements  of  the  moment. 

The  future  of  CAI  and  TAI  is  also  discussed.  Given  a  number 
of  assumptions,  it  is  predicted  that  in  less  than  10  years,  CAI 
costs  will  be  roughly  half  the  cost  of  comparable  TAI. 


Walker,  G.  A.  Life  Cycle  Cost/System  Effectiveness  Evaluation 
and  Criteria.  The  Boeing  Company,  Document  No.  DIbO- 1  76k&'-'l  (Un- 
classified),  January  197*»,  (AD  916  001).  This  report  contains  the 
results  of  an  independent  research  and  development  task  on  life 
cycle  cost  performed  by  Boeing  Aerospace  Company.  Oue  to  the 
growth  of  weapon  acquisition  and  support  costs,  a  need  is  recog¬ 
nized  for  a  cost  analysis  technique  that  is  simple,  flexible,  low 
cost,  and  easily  applied  throughout  early  phases  of  a  program. 
Literature  on  life  cycle  cost  technology  was  reviewed  and  organized. 
The  conclusions  were  that  no  LCC  model  had  been  successfully  applied 
to  perform  total  LCC  trade-off  studies  on  a  major  on-going  program, 
and  that  a  practical  method  for  determining  LCC  does  not  yet  exist. 
Also,  the  report  notes  that  no  attempt  has  been  made  to  develop  a 
standard  tool,  or  technique,  for  determining  "systems  worth". 
Recommendations  are  offered  for  improving  methods  of  effectively 
performing  LCC  analysis  on  new  design  programs. 

A  discussion  follows  on  program  life  cycle  phases;  how  cost 
analysis  applies  to  each  of  these  phases;  how  model s/ana 1 ys i s 
techniques  should  be  selected;  functional  elements  to  be  considered; 
and  general  documentat ion  pertaining  to  the  research  project. 


Shrivcr,  E.  L.  and  Hart,  F.  L.  Study  and  Proposal  for  the 
Improvement  of  Military  Technical  Information  Transfer  Methods. 

U.  S.  Army  Human  Engineering  Laboratory,  Technical  Memorandum 
29-75  (Unclassified),  December  1975,  (AD  A023  ^09).  The  study  pro¬ 
ceeds  from  the  observation  that  the  cost  of  owning  equipment  is 
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more  than  the  cost  of  procuring  it.  The  objective  of  the  study 
was  to  develop  a  more  cost  effective  equipment  maintenance  pro¬ 
gram.  It  reviewed  new  concepts  for  technical  manuals  which  have 
been  developed  and  tested  over  the  past  twenty  years.  The  pro¬ 
cesses  and  techniques  used  by  these  concepts  are  identified, 
analyzed,  and  summarized  in  a  specification  incorporating  the  best 
features  of  each.  The  improved  maintenance  program  developed  by 
this  study  could  result  in  lower  maintenance  costs,  better  relia¬ 
bility,  and  higher  availability  of  Army  material. 

The  report  is  organized  into  seven  tasks: 

Task  1:  To  identify  new  concepts  for  maintenance  that  hold  pro¬ 
mise  for  reducing  the  cost  of  equipment  ownership. 

Task  2:  To  analyze  the  new  concepts  in  order  to  identify  the 

fundamental  elements  responsible  for  their  effect  upon 
maintenance  cost. 

Task  3:  To  match  fundamental  elements  to  particular  Army  situations, 
to  evaluate  their  validity,  and  to  make  broad  cost  pro¬ 
jections  . 

Task  *»:  To  review  the  fundamental  elements  against  the  Commodity 
Systems,  and  project  the  cost  of  ownership  which  could 
accrue  through  use  of  these  elements  to  produce  all  Army 
TMs. 

Task  5:  To  prepare  a  specification  which  includes  all  the  fun¬ 
damental  elements  tailored  to  Army  situations  where  they 
are  appropriate. 

Task  6:  Prepare  a  sample  manual  based  on  the  specification. 

Task  7:  Prepare  a  test  plan  for  testing  the  sample  material. 

Instead  of  using  training  to  provide  the  new  maintenance  personnel 
with  general  information  which  is  combined  with  specific  equipment 
descriptions  on  the  job,  the  recommended  approach  is  to  analyze 
all  jobs  and  describe  exactly  what  is  to  be  done  in  the  TM  to  main¬ 
tain  the  equipment. 

Literature  Dealing  with  Efficiency  Issues.  Roscoe,  S.  N. 
Incremental  Transfer  Effectiveness,  Aviation  Research  Laboratory 
at  University  of  Illinois,  Technical  Report  ARL-70-5/AFOSR-70-1 
(Unclassified),  February  1971,  (AD  7^8  235)  and  Roscoe,  S.  N. 
Incremental  Transfer  Effectiveness,  Human  Factors,  1971,  13(6), 
pp.  561-567,  (AD  7^9  121).  These  two  reports  present  a  method  for 
measuring  transfer  of  learning.  They  deal  with  a  method  for  deter- 
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mining  when  to  substitute  one  type  of  training  for  another  based 
on  the  incremental  time  saved  and  cost  incurred. 


A  universal  relationship  is  postulated  in  which  the  effec¬ 
tiveness  of  successive  increments  of  training  on  one  task,  as 
measured  by  the  relative  incremental  savings  in  learning  a  second 
task,  is  a  negatively  decelerated  function  of  the  time  devoted  to 
pretraining  or  interpolated  training  on  the  former  task.  It  is 
further  postulated  that  the  relationship,  inferred  from  aircraft- 
pilot  training  situations,  applies  in  different  forms  to  all  edu¬ 
cational  experiences,  thereby  allowing  all  formal  educational 
programs,  in  theory,  to  be  evaluated  in  terms  of  their  incremental 
cost  effectiveness. 


Braby,  R.,  et.  al.  Staff  Study  on  Cost  and  Training  Effec¬ 
tiveness  of  Proposed  Training  Systems,  Training  Analysis  and  Eval¬ 
uation  Group (TAEG)  at  Naval  Training  Equipment  Center,  TAEG  Re¬ 
port  1  (Unclassified),  1972.  This  report  discusses  the  results  of 
a  staff  study  to  develop  a  procedure  for  evaluating  alternate 
training  systems  and  for  selecting  among  them  the  most  cost  effec¬ 
tive  ones.  A  prototype  Training  Effectiveness  and  Cost  Effective¬ 
ness  Prediction  Model  (TECEP)  was  developed  and  tested.  This  is 
a  method  for  the  optimization  of  training  media  allocation  on  the 
basis  of  fixed  training  effectiveness  and  minimum  cost.  The  over¬ 
all  model  consists  of  three  sub-models:  media  substitution  and 
selection,  life  cycle  cost,  and  optimization.  Media  alternatives 
are  identified  through  the  use  of  a  media  selection  matrix.  Media 
costs  are  developed  through  the  use  of  life  cycle  cost  analysis 
(categorization  of  cost  factors,  estimation  of  cost  factor  values, 
and  calculation  of  cost  per  utilization  hour  for  each  medium). 
Linear  programming  is  used  as  an  optimizing  tool  for  determining 
least  cost  training  media.  Application  of  the  TECEP  Model  is 
mad~  to  the  TA-4  Aircraft  Training  System  as  a  demonstration  of 
the  feasibility  of  the  model. 


Spangenberg,  R.  W.,  et.  al .  The  State  of  Knowledge  Pertaining 
to  Selection  of  Cost-Effective  Training  Methods  and  Media.  Human 
Resources  Research  Organizotion,  Technical  Report  7 3 — 1 3  (Unclassi¬ 
fied),  June  1973.  (AD  76319*0.  In  order  to  prepare  a  plan  for  re 
search  toward  empirical  determination  of  criteria  and  procedures 
for  optimal  selection  of  cost-effective  methods  and  media,  a  re¬ 
view  and  analysis  of  pertinent  literature  was  conducted.  The  re¬ 
search  fell  into  two  major  categories: 

o  those  pertaining  to  methods-media  definition  and 
classification,  and  methods-media  selection  criteria 
and  procedures 


o  those  pertaining  to  training  cost-effectiveness  and 
analytical  procedures. 


The  authors  reported  that  the  empirical  data  found  on  the 
relative  cost-effectiveness  of  methods  and  media  were  insufficient 
as  a  basis  for  reliable  selection  of  methods  and  media  for  specific 
training  tasks.  Also,  the  existing  methods-media  selection  pro¬ 
cedures,  training  cost-analysis  procedures,  and  suggested  approaches 
for  developing  such  procedures  were  inadequate  for  Army  needs,  al¬ 
though  portions  of  some  existing  procedures  may  be  useful  in  de¬ 
veloping  new  procedures  for  Army  use.  Most  selection  procedures 
developed  outside  the  Army  were  designed  for  use  in  civilian  schools, 
and  others  were  based  upon  specific  restrictive  assumptions  and 
theories  that  do  not  sufficiently  apply  to  Army  conditions.  They 
found  that  existing  selection  procedures  used  in  civilian  schools 
have  proved  to  be  unsuccessful  even  for  their  intended  purposes 
(selection  criteria  used  were  too  general).  New  approaches  to 
methods-media  problems  are  also  discussed. 


Braby,  R.,  et.  al .  A  Technique  for  Choosing  Cost-Effective 
Instructional  Delivery  Systems,  Training  Analysis  and  Evaluation 
Group  at  Naval  Training  Equipment  Center,  TAEG  Report  No.  16 
(Unclassified),  April  1975.  The  purpose  of  this  report  is  to  make 
available  to  training  specialists  a  procedure  for  choosing  instruc¬ 
tional  delivery  systems  appropriate  to  various  types  of  military 
training.  The  procedure  is  the  Training  Effectiveness,  Cost  Ef¬ 
fectiveness  Prediction  (TECEP)  technique.  It  is  intended  to  pro¬ 
vide  an  orderly  approach  for  the  skilled  training  system  designer 
to  use  in  making  delivery  system  choices  during  the  conceptual 
design  phase. 

A  three-step  procedure  is  described  in  which  training  ob¬ 
jectives  are  classified  and  organized  into  groups,  media  capable 
of  supporting  these  strategies  are  identified,  and  the  costs  of 
alternative  forms  of  training  ore  projected.  Using  such  infor¬ 
mation,  it  is  contended  that  optimum  delivery  system  choices  can 
be  made. 

The  TECEP  technique  is  viewed  as  a  single  part  of  larger 
Training  System  Development  Model.  The  technique  is  not  mechanical 
and  does  require  user  expertise.  It  is  best  described  as  a  job  aid 
for  an  experienced  training  system  designer.  The  report  suggests 
that  it  can  provide  a  pathway  and  procedures  for  systematically 
coming  to  grips  with  critical  issues  in  planning  cost-effective 
instruction. 

Analyzing  Training  Effectiveness,  Headquarters  U.  S.  Army 
Training  and  Doctrine  Command,  TRADOC  PAM  71-8  (Unclassified), 
December,  1975.  This  document  is  primarily  didactic.  It  describes 
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methods  used  by  TRADOC  to  point  the  way  toward  increased  effec- 
tiveness  for  selected  combat  material  through  improved  training. 

The  methods  discussed  include  problem  formulation,  model  con¬ 
struction,  and  model  testing;  as  well  as  illustrations  of  model 
use.  The  first  chapter  discusses  the  motivation,  need,  and  pay¬ 
off  for  improved  training.  The  second  chapter  summarizes  a  num¬ 
ber  of  useful  operations  research  (OR)  models,  including  a  gener¬ 
al  effectiveness  model,  learning  curves,  readiness  models,  and 
simulation  models.  Analytical  tools  and  procedures  are  discussed 
in  the  next  chapter.  Tools  include  systems  analysis  and  task 
analysis;  procedures  include  the  Train  Up  Study  (TUS) ,  the  CTEA, 
and  the  Training  Input  To  COEA  study.  The  final  chapter  iden¬ 
tifies  several  of  the  major  funding,  research,  testing  and  analy¬ 
tic  resources  that  are  available  to  support  training  effectiveness 
analysis.  Programs  discussed  include  the  Concept  Evaluation  Pro¬ 
gram  (CEP) ,  the  Human  Resources  Research  Program  (HRRP) ,  and  the 
TRADOC  Study  Program  (TSP) .  Various  test  organizations  are  pro¬ 
filed,  as  is  TRASANA. 

Toomepuu,  J.  Army  Flight  Simulator  Programs  from  the  User's 
Viewpoint.  U.  S.  Army  Training  Support  Center.  (Unclassified)  , 

May  1976,  (AD  A029  266).  The  report  discusses  problems  and  desira¬ 
ble  directions  in  training  effectiveness  analysis.  It  covers  three 
topics : 

o  user  evaluation  of  the  quality  and  suitability  of 
the  Army  flight  simulator  program 

o  cost  effectiveness  of  flight  simulators 

o  research  initiatives  needed  to  meet  Army  aviation 
training  needs. 

The  author  contends  that  while  operational  testing  is  the 
primary  means  for  user  evaluation,  some  evaluation  is  needed  be¬ 
fore  this  point  is  reached.  User  participation  should  be  provided 
for  all  stages  of  the  research  program.  Causes  for  lack  of  user 
participation,  as  well  as  suggestions  for  user  participation,  are 
discussed. 

Regarding  the  effectiveness  measurement  of  simulators,  the 
author  argues  that  a  methodology  is  needed  that  is  based  on  task 
analysis,  identifies  the  cues  required  by  pilots  to  perform  these 
tasks,  and  matches  cue  elements  with  the  elements  of  the  simulator 
subsystems  that  provide  the  cues.  A  training  cost  allocation 
methodology  also  needs  to  be  developed  to  adequately  measure  both 
alternative  training  system  costs  and  simulator  training  costs, 
and  the  contribution  to  the  overall  cost  by  simulator  subsystems 
and  their  elements.  Simulators  should  be  justified  on  the  basis 
of  all  relevant  cost-effectiveness  considerations. 


Research  initiatives  in  the  area  of  flight  simulators  and 
their  users  are  proposed  in  the  context  of  future  aircraft  and 
organizations,  operational  doctrine  and  tactics,  and  management 
of  aviation  training  assets. 


Lukasczyk,  N.  Efficiency  Indicators  for  Education  and 
Training.  Naval  Postgraduate  School,  Master's  Thesis  (Unclas¬ 
sified),  June  1976,  (AO  A028  854).  The  purpose  of  this  report 
is  to  propose  and  analyze  indicators  of  education  and  training 
efficiency  for  the  Chief  of  Naval  Education  and  Training  (CNET) . 
The  indicators  should  measure  how  effectively  CNET  is  producing 
a  given  output  in  a  given  time  period  relative  to  previous  time 
periods.  Three  indicators  are  discussed:  the  Staff/Student 
Ratio,  Cost  per  Student  per  Unit  Time,  and  Cost  per  Graduate. 

The  arguments  contend  that  the  cost  per  graduate  is  the  most 
appropriate  indicator  for  a  single  course.  Procedures  are  derived 
for  the  appropriate  aggregation  of  multiple  courses.  The  derived 
indicators  have  the  mathematical  form  of  the  Laspeyres  and  Paasch 
indicators,  used  in  economic  theory  for  the  cost  of  living  index. 
They  are  applied  to  sixty  courses  of  SSC  San  Diego  and  compared 
to  indicators  determined  by  linear  regression  based  on  the  same 
data  set.  The  resulting  values  of  the  indicators  are  believed 
to  be  helpful  to  locate  the  area  of  interest  and  detail  for  fur¬ 
ther  decision  making. 


Cost  and  Training  Effectiveness  Analysis  (CTEA)  of  the  CH-47 
FI ight  Simulator  (CH47FS) ,  CH47FS  Study  Group  at  U.  S.  Army  Aviation 
Center,  ACN  23879  (Unclassified),  December  1976,  (AD  A033  972)  and 
Toomepuu,  J.  Test  and  Evaluation  of  the  Army's  CH-47  Helicopter 
FI  ight  Simulator ,  U.  S.  Army  Training  Support  Center,  (Unclassified), 
January  1977,  [AD  AO36  159).  Both  reports  concern  the  determination 
of  the  cost  and  training  effectiveness  of  the  CH-47  Flight  Simu¬ 
lator  when  utilized  in  the  CH-47  Aviator  Qualification  Course  and 
CH-47  aviator  unit  training. 

The  objectives  were  to: 

o  determine  the  training  effectiveness  of  each  alter¬ 
nate  training  package 

o  estimate  the  pertinent  costs  of  each  alternate 
training  package 

o  determine  the  cost  and  training  effectiveness  of 
each  alternate  training  package 

o  rank  order  the  alternate  training  packages  on  the 
basis  of  appropriate  quantitative  cost  and  training 


effectiveness  measures,  and  judgmental  evaluations 
of  the  situations  in  which  the  alternates  are  ex¬ 
pected  to  be  used. 

o  prepare  a  recommended  CH47FS  basis-of-issue  plan 
for  the  preferred  packages 

o  ascertain  the  impact  each  alternate  training  pack¬ 
age  will  have  upon  Army-wide  combat  readiness  of  Army's 
CH-47  assets. 

The  scope  of  the  study  vas  to: 

o  address  the  cost  and  training  effectiveness  of  the 
alternate  training  packages,  considering  anticipated 
resource  constraints,  combat  readiness  of  the  Army's 
CH-A7  assets,  and  safety 

o  include  a  review  of  the  Synthetic  Flight  Training 
System  Training  Device  Requirement 

o  recommend  a  basis-of-issue  plan  for  the  CM7FS. 


The  methodology  includes  variable  cost/fixed  effectiveness 
analysis  and  judgmental  evaluations,  effectiveness  analysis,  and 
cost  analysis.  Measures  of  training  effectiveness  used  were 
aviator  performance  evaluation  scores  and  training  hour  necessary 
to  attain  specific  performance  objectives.  The  life  cycle  cost 
estimate  was  determined  in  terms  of  training  cost  per  aviator  for 
each  alternate,  and  total  cost  for  each  alternate. 


A  Survey  and  Analysis  of  Military  Computer-Based  Training 
Systems:  (A  Two-Part  Study)  Volume  II,  McDonnell  Douglas  Astro¬ 

nautics  Company  -  East,  MDC  E 1 5 70  (Unclassified),  May  1977,  (AD 
A0<«3  358).  The  purpose  of  this  report  was  to  develop  a  means 
for  assessing  the  cost  versus  expected  benefits  of  innovations 
in  computer-based  training  systems.  A  three  pronged  approach 
was  used  to  address  the  problem.  First,  a  descriptive  model  of 
a  generalized,  computer-based  training  system  was  developed. 
Second,  a  predictive  model  was  generated  to  describe  student  per¬ 
formance  in  a  computer-based  training  environment.  Third,  a  com¬ 
puter  program  incorporating  the  predictive  model  was  developed 
to  estimate  the  cost  of  implementing  one  particular  training  in¬ 
novation,  i.  c.,  revision  of  a  conventional  instructional  pro¬ 
gram  into  a  self-paced  instructional  program.  A  linear  prediction 
model  was  developed  to  predict  the  time  required  for  a  student 
to  complete  a  unit  of  self-paced  instruction  thereby  demonstrating 
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the  time  saved  by  converting  from  a  conventional  instructional 
mode.  This  was  combined  with  the  Rand  Corporation's  MODIA  cost 
analysis  program.  The  study  demonstrated  that  a  viable  time 
savings  and  cost  model  can  be  developed  given  a  sufficient  data 
base  from  which  to  draw  the  necessary  course  content  and  student 
characteristics  descriptions.  The  generality  and  precision  of 
such  a  model  is  dependent  on  the  breadth  and  accuracy  of  the 
data  base  from  which  the  model  is  derived.  Although  the  data 
base  employed  in  this  study  was  not  as  complete  as  could  be  de¬ 
sired,  the  resultant  model  was  found  to  be  reasonably  accurate 
and  is  sufficiently  general  to  be  of  value  in  the  evaluation  of 
courses  for  individualization.  Thus,  a  methodology  was  developed 
and  demonstrated  which  can  become  a  useful  tool  as  additior-al 
data  are  accumulated  from  computer-based  training  systems. 

B-33  Cost  and  Training  Effectiveness  Analysis  Report,  U.  S. 
Army  TRADOC  Systems  Analysis  Activity,  TRASANA  1577  (Unclass if ied) 
June,  1977*  This  report  concerns  the  cost  and  training  effective¬ 
ness  of  the  Swedish  SAAB-SCANIA  BT-33  simulator  (for  training  for¬ 
ward  observers)  as  compared  to  current  methods  of  training  (field 
and  live  fire  exercises).  The  objectives  of  the  analysis  are  to 
determine  the  effectiveness  of  the  BT-33  to  train  forward  obser¬ 
vers  in: 

o  preparation  of  the  call  for  fire 
o  target  location 
o  fire  adjustment 

In  addition,  it  was  to  determine  the  relative  costs  of  three  dif¬ 
ferent  combinations  of  BT-33  simulation  and  conventional  forward 
observer  training.  The  methodology  consisted  of  defining  alter¬ 
natives,  making  explicit  assumptions,  and  determining  the  costs 
for  each  alternative.  Sensitivity  of  alternative  cost  rankings  was 
then  conducted. 

The  report  concludes  that  the  BT-33  proved  to  be  an  effective 
training  device  to  complement  classroom  training.  It  appeared  to 
offer  significant  savings  compared  to  conventional  live  fire  train¬ 
ing  methods;  however,  it  results  in  increased  costs  if  used  only 
for  conventional  classroom  training. 

Cost  and  Training  Effectiveness  Analysis  Handbook,  U .  S . 

Army  TRADOC  Systems  Analysis  Activity,  TRASANA  Draft  Document, 

July,  1976.  The  purpose  of  the  document  is  to  provide  guidance 
and  assistance  to  TRADOC  personnel  in  conceptualizing,  planning, 
and  conducting  cost  and  training  effectiveness  analyses  (CTEAs)  . 
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A  CTEA  is  defined  as  a  methodology  which  involves  documented 
investigation  of  the  comparative  effectiveness  and  costs  of 
alternative  training  systems  for  attaining  defined  performance 
objectives,  taking  into  account  usage  patterns  and  training 
scenarios.  Three  types  of  CTEAs  are  identified:  the  Train- 
up  Study  (TUS) ,  the  Training  Analysis  for  COEA  (TAC) ,  and  the 
Training  Development  Study  (TUS).  Chapters  l  and  2  present  an 
introduction  and  overview  of  the  CTEA  process.  The  remaining 
chapters  treat,  on  a  step-by-step  basis,  the  general  procedures 
comprising  a  CTEA  study.  These  procedures  include  identifying 
the  performance  objectives,  planning  the  analysis,  determining 
the  effectiveness  of  the  training  program  under  study,  esti¬ 
mating  the  true  costs  of  the  program,  and  finally  comparing  the 
effectiveness  and  costs. 


Literature  Dealing  with  Valuation  Issues.  Barsby,  S.  L. 
Cost-Benefit  Analysis  and  Manpower  Programs,  Lexington  Books, 
Lexington,  Massachusetts:  1972.  A  number  of  cost-benefit 
analyses  as  they  relate  to  manpower  programs  are  discussed  in 
this  volume.  The  general  categories  into  which  they  are  divided 
are  vocational  education  (secondary  and  post-secondary  schools), 
institutional  out-of-school  retraining  programs  under  state  and 
federal  legislation,  and  programs  with  an  OJT  (On  the  Job  Train¬ 
ing)  component  and  miscellaneous  programs. 

Five  studies  concerning  vocational  education  in  secondary 
schools  and  two  studies  on  post-secondary  schools  arc  examined. 

In  general,  relevant  costs  included  operating  expenses,  foregone 
earnings,  direct  costs  borne  by  students,  and  tax  adjustments  for 
unpaid  property  taxes.  Benefits  were  typically  defined  as  dis¬ 
counted  future  earnings. 

Studies  on  institutional  retaining  programs  are  divided  into 
those  concerning  state  or  Area  Redevelopment  Act-sponsored  pro¬ 
grams  and  more  recent  studies  evaluating  Manpower  Development  and 
Training  Act-sponsored  programs.  Four  studies  in  each  area  are 
reviewed.  Generally,  costs  include  operating  costs,  a  portion  of 
state  and  federal  administration  costs  of  overall  ARA  activities, 
transfer  payments  made  to  trainees  while  participating  in  train¬ 
ing,  opportunity  costs,  and  direct  costs  to  students.  Benefits 
wore  calculated  as  discounted  future  wages. 

The  last  section  discusses  OJT  and  miscellaneous  programs. 
Studies  that  are  examined  concern  the  Neighborhood  Youth  Corps, 
Work  Experience  and  Training  Program,  MDTA-sponsored  OJT,  Job 
Corps,  Bureau  of  Indian  Af fa i rs-sponsored  retraining,  dropout- 
prevention  program,  high  school  job  placement  services,  and  a 
vocational  rehabilitation  program. 
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Harberger,  A.  C.,  et.  al .  (eds.)  Benefit  Cost  Analysis  1971- 
Aid i nc-Atherton ,  Chicago:  1972.  This  volume  contains  four  papers 
of  interest  concerning  investment  in  human  capital,  a  phrase  synon¬ 
ymous  with  training. 

•  "A  Benefit-Cost  Analysis  of  the  Upward  Bound  Program"  (W.  I. 
Garms)  is  an  evaluation  of  that  project,  operated  by  the  U.  S.  Of¬ 
fice  of  Education.  Individual  benefits  were  defined  to  include 
increased  lifetime  income  after  taxes,  the  stipend  received  while 
in  the  program,  and  scholarships  and  grants  received  while  in  col¬ 
lege.  Individual  costs  were  defined  as  the  college  tuition,  extra 
living  costs  associated  with  attending  college,  foregone  earnings 
while  attending  high  school  and  college,  and  foregone  unemployment 
and  welfare  payments  as  a  result  of  being  more  fully  employed  dur¬ 
ing  his  working  lifetime.  Social  benefits  are  lifetime  income 
differentials  of  those  enrolled  in  the  program.  "Unemployment  and 
welfare  payments  are  transfer  payments  from  the  social  viewpoint. 
Since  they  are  included  in  the  income  series  as  a  reduction  in  net 
income  differentials,  their  amount  must  be  estimated  and  added  back 
to  the  differentials."  The  social  costs  considered  are  the  direct 
cost  of  the  program  to  the  government,  less  the  amount  paid  in  sti¬ 
pends;  cost  of  the  program  to  par t i ciapt i ng  colleges,  cost  of  edu¬ 
cating  the  students  in  the  program,  both  in  high  school  and  in  col¬ 
lege;  extra  living  costs  of  students  while  in  school;  and  foregone 
income.  Experimental  and  control  groups  were  used  in  the  empirical 
analysis.  Results  of  the  evaluation  were  not  clearcut,  tending  to 
support  both  the  positive  and  negative  positions  on  the  value  of  the 
Upward  Board  Program. 

•  "Economic  Returns  to  Vocational  and  Comprehensive  High  School 
Graduates"  (T.  Hee,  et.  al.)  is  a  comparison  of  vocational  and 
comprehensive  secondary  education,  comparing  costs  and  labor  mar¬ 
ket  performances  (in  terms  of  earnings  and  employment)  among  voca¬ 
tional  and  comprehensive  high  school  graduates  who  did  not  attend 
college.  Multiple  regression  analysis  is  used  on  questionnaire 
data.  The  authors  conclude  that  "the  monetary  returns  of  voca¬ 
tional-technical  graduates  are  higher  than  those  of  comprehensive 
graduates ." 

•  "Criteria  for  Public  Investment  in  the  Two-Year  College:  A 
Program  Budgeting  Approach"  (Harry  N.  Heinemann  and  Edward  Sussna) 
is  a  cost-benefit  analysis  of  education  in  a  two-year  college. 

"Cost  projections  reflect  estimates  of  size  of  student  body,  the 
ratio  of  full-  to  part-time  students,  and  expenditures  on  tuition 
and  supplies  by  students,  and  operating  costs  and  plant  and  equip¬ 
ment  outlays  from  public  funds.  To  these  direct  outlays  must  be 
added  the  foregone  income  of  full-time  students."  Opportunity 
costs  are  foregone  imputable  to  a  two-year  school.  Internal  rate 
of  return  calculations  were  carried  out,  indicating  a  return  to 
students  of  approximately  \S%  on  their  investment. 
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•  "The  MITCE  Project  as  Judged  by  Efficiency  and  Distributional 
Investment  Criteria"  (David  0.  Sewell)  concerns  a  cost/bencfit 
analysis  of  the  MITCE  (Manpower  Improvement  Through  Conmunity 
Effort)  project.  Benefits  used  were  discounted  incremental  earn¬ 
ings.  Costs  included  were  the  direct  costs  of  training  associated 
with  the  daily  operation  of  courses,  indirect  costs  incurred  by 
federal  and  state  agencies  in  program  administration,  and  oppor¬ 
tunity  costs  (foregone  earnings).  Using  present  value  calcula¬ 
tions,  the  author  concludes  that  "it  thus  appears  likely  that  the 
MITCE  project  sat i s f i es . . . the  minimum  condition  of  acceptability 
for  an  antipoverty  program:  that  it  should  raise  the  incomes  of 
the  poor." 

Gay ,  R.  M.,  et.  a  1 . ,  Cost  and  Efficiency  in  Military  Specialty 
Training,  Rand  Corporation,  Rand  Paper  Series  P-5160  (Unclassified) 
January  1 97^ ,  (AD  786  652).  This  report  deals  with  the  evaluation 
of  specialty  training  for  first-term  enlisted  personnel.  The 
methodology  considered  costs  of  both  formal  and  on-the-job  training 
as  well  as  the  returns  to  training  for  first-term  enlisted  person¬ 
nel.  Training  is  treated  as  an  investment,  and  the  present  value 
method  is  used  for  the  evaluation.  Net  return  on  training  is  the 
difference  between  the  present  values  of  benefits  and  expenditures. 
Efficiency  is  defined  as  "the  set  of  training  practices  which  maxi¬ 
mizes  the  net  return  to  training."  Estimation  of  on-the-job  costs 
are  made  for  specific  individuals  rather  than  for  the  "typical 
trainee."  The  trainee's  net  contribution  to  his  unit  over  tine 
is  compared  with  his  cost  to  the  military  over  tine.  A  method  for 
measuring  the  value  of  the  marginal  product  of  the  individual  is 
suggested.  The  authors  define  the  benefit  derived  by  the  military 
from  the  training  of  personnel  as  the  value  of  the  labor  services 
provided  by  the  individual  in  each  period.  This  method  was  tested 
on  the  Aircraft  Maintenance  Specialists  at  Norton  Air  Force  Base, 
California.  The  study  appeared  to  support  the  feasibility  of  the 
model . 


Haveman,  R.  H.,  et.  al .  (eds.)  Benefit-Cost  and  Policy  Analysis 
1 973 ,  Chicago:  Aldine  Publishing  Company,  197A.  The  volume  con¬ 
tains  three  articles  that  are  relevant  to  the  valuation  of  training 
insofar  as  they  deal  with  the  measurement  and  evaluation  of  invest¬ 
ment  in  human  capital. 

•  "Social  Returns  to  Quantity  and  Quality  of  Schooling:  (G.  E. 
Johnson  and  F.  P.  Stafford)  estimates  the  marginal  social  rates  of 
return  to  quality,  as  well  as  quantity,  of  schooling.  With  quali¬ 
ty  measured  as  expenditure  per  pupil  per  unit  time,  the  determinants 
of  hourly  wage  rates  were  determined  using  regression  analysis. 

"The  authors  find  high  but  diminishing  marginal  returns  to  invest¬ 
ment  in  (the  quality  of  education)." 


•  "A  Statistical  Analysis  of  the  0E0  Experiment  in  Educational 
Performance  Contracting"  (I.  Garfinkel  and  E.  M.  Gramlich)  deals 
with  the  hypotheses  that  private  contractors  can  be  more  effec¬ 
tive  in  improving  the  educational  status  of  disadvantaged  stu¬ 
dents  than  the  public  school  system.  The  paper  reports  on  an 
analysis  of  test  data  from  an  experiment  sponsored  by  the  Office 
of  Economic  Opportunity  in  performance  contracting.  Experimental 
and  control  groups  were  used,  and  Achievement  score  gains  of  both 
groups  were  compared.  Two  methodological  problems  (imperfect 
matching  of  experimental  and  control  students  and  random  measure¬ 
ment  error)  are  discussed.  Results  indicated  that  the  performance 
contractors  were  not  able  to  improve  the  status  of  disadvantaged 
chi ldren. 

•  "Measuring  Benefits  from  Prison  Reform:  (J.  Holahan)  dis¬ 
cusses  the  measurement  of  prison  reform  and  applies  the  model  to 
an  experimental  rehab i l i tat i ve  program.  "The  benefits  to  society 
include: 

o  a  reduction  in  forced  redistribution  of  wealth, 
physical  injury  and  property  damage 

o  a  lower  level  of  public  expenditures  on  the  criminal 
justice  system 

o  increased  productivity/earnings  and  higher  employ¬ 
ment  rates  over  time 

o  reduction  in  other  social  costs  of  crime  such  as 
private  crime  deterrence  expenditures,  migration, 
avoidance  of  normal  activity,  etc." 

"The  cost  of  recidivism  is  this  model  depends  on  the  direct  cost 
of  crimes  (stolen  or  damaged  property,  personal  injury,  etc.), 
the  costs  of  police  services,  the  probability  of  dismissal  or  of 
specific  types  of  trials,  the  costs  of  judicial  process,  the  pro¬ 
bability  of  a  sentence  and  of  specific  types  of  sentences  and  the 
costs  of  the  correctional  system.  It  further  depends  on  the  pro¬ 
bability  of  recidivism  and  the  additional  costs  that  would  in¬ 
volve."  The  focus  of  the  analysis  is  on  property  crime.  Results 
are  inconclusive. 


ECONOMIC  LITERATURE 


The  economic  evaluation  literature  search  revealed  a  very  ex¬ 
tensive  set  of  monographs  and  journal  articles  dealing  with  the 
general  principles  of  program  appraisal.  An  exhaustive  biblio- 
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graphy  in  this  area  might  easily  include  upwards  of  1,000  entries 
Time  and  resources  did  not  permit  a  review  of  even  a  tenth  of  that 
literature.  Nonetheless,  by  reviewing  survey  articles  and  mono¬ 
graphs  as  well  as  recognized  "classics",  we  are  confident  that 
little  of  remarkable  substance  has  escaped  our  attention. 

Currie,  J.  M.,  Murphy,  J.  A.  and  Schmitz,  A.  The  Concept 
of  Economic  Surplus  and  Its  Use  in  Economic  Analysis.  The  Eco¬ 
nomic  Journal,  December,  1971.  The  authors  identify  the  concept 
of  economic  surplus  as  originating  with  Dupuit  (1968)  and  Marshall 
(1930)  and  refined  by  Hicks  (19^0-41,  19^3,  19*»6,  19^5-^6.  1956) 
and  Henderson  (^^O-^l).  The  four  measures  of  consumers'  surplus 
defined  by  Hicks  include  compensating  variation,  compensating 
surplus,  equivalent  variation,  and  equivalent  surplus.  The  "sur¬ 
plus"  area  beneath  the  Hicksian  compensated  demand  curve  is  an 
exact  measure  of  the  compensating  variation.  Henderson's  posi¬ 
tion  is  reported  as  being  that  "for  any  specific  question  only 
one  of  these  measures  will  usually  be  relevant."  Mishan's  (19^7“ 
*»8)  position  is  reported  as  being  that  "only  compensating  varia¬ 
tion  and  equivalent  variation  should  be  considered  .  .  .  'in  all 
plausible  ci rcunstances . '  "  Applications  of  economic  surplus  to 
issues  of  resource  misal locat ion  and  international  trade  are  dis¬ 
cussed  . 


Prest,  A.  ft.  and  Turvey,  R.  Cost-Benefit  Analysis:  A  Sur¬ 
vey.  The  Economic  Journal.  December,  1965.  "Cost-benefit  analy 
sis  (or  'investment  planning'  or  'project  appraisal')  is  a  prac¬ 
tical  way  of  assessing  the  desirability  of  projects,  where  it  is 
important  to  take  a  long  view  .  .  .  and  a  wide  view  .  .  .,  it  im- 
Pl  ies  the  enumeration  and  evaluation  of  all  the  relevant  costs 
and  benefits."  The  authors  trace  the  history  of  cost-benefit 
analysis  beginning  with  Dupuit's  18^4  paper  through  the  (then) 
present  applications  to  water  resources  and  other  practical  areas 
In  an  extensive  section  on  general  principles,  Prest  and  Turvey 
identify  some  of  the  questions  arising  in  attempting  to  perform 
a  cost-benefit  study,  including: 

o  "Which  costs  and  which  benefits  are  to  be  included? 

o  "How  are  they  to  be  valued? 

o  "At  what  interest  rate  are  they  to  be  discounted? 

o  "What  arc  the  relevant  constraints?" 

Among  the  applications  areas  surveyed  by  the  authors  are  water, 
transport,  land  use,  health,  and  education  projects. 


McKean,  R.  N.  Efficiency  in  Government  Through  Systems 
Ana  lysis.  John  Wiley  and  Sons,  Inc.,  New  York:  1958.  This 
volume  is  one  of  the  seminal  works  in  general  public  investment 
evaluation,  although  its  overt  focus  is  the  evaluation  of  water 
resource  projects.  Topics  addressed  include  discussion  criteria, 
choice  of  alternatives,  treatment  of  intangibles  and  uncertainty, 
externalities,  double  counting,  and  secondary  effects.  Two  case 
studies  in  water  resource  investment  evaluation  are  also  pre¬ 
sented. 


Samuelson,  P.  A.  Foundations  of  Economic  Analysis.  Athe- 
neum.  New  York:  19&5.  (originally  published  by  Harvard  Univer¬ 
sity  Press,  19^7).  This  work  is  one  of  the  modern  classics  in 
economics  showing  the  unifying  mathematical  structure  under¬ 
lying  the  various  branches  of  economic  theory.  Topics  treated 
include  equilibrium  systems,  theory  of  optimization,  theory  of 
cost  and  production,  theory  of  consumer  behavior,  welfare  eco¬ 
nomics,  stability  of  equilibrium,  and  dynamical  theory.  In  the 
author's  treatment  of  some  speciat  aspects  of  consumer  theory, 
he  discusses  six  different  measures  of  consumers'  surplus,  in¬ 
cluding  the  compensating  and  equivalent  budget  variations. 

Lesournc,  J.  Cost-Benefit  Analysis  and  Economic  Theory. 

North  Holland  Publishing  Company,  Amsterdam:  1975-  This  mono¬ 
graph,  a  translation  from  the  French,  is  divided  into  two  major 
parts:  theory,  and  applications.  The  author  distinguishes  be¬ 

tween  the  theory  of  the  economic  optimum,  which  he  calls  welfare 
theory;  and  the  theory  of  the  comparison  of  economic  states,  which 
he  calls  cost-benefit  analysis.  Starting  from  a  basic  model  of 
the  economy  which  exhibits  one  time  period,  no  state,  perfect 
prediction,  given  information  and  cost-free  decisions,  Lesourne 
discusses  the  analysis  of  marginal  and  structural  transformations, 
taxation  and  extensions  to  multi-period  models,  imperfect  pre¬ 
diction  and  non-cost-free  information.  The  areas  of  applications 
of  cost-benefit  analysis  discussed  include  the  industrial  sec¬ 
tor,  the  transport  sector,  urban  planning,  location  and  regional 
planning,  public  health,  education,  and  general  administration 
dec i s ions . 


Sassone,  P.  G.  and  Schaffer,  W.  A.  Cost-Benefit  Analysis: 

A  Handbook.  Academic  Press,  New  York:  1 978 .  This  monograph 
presents  theoretical  treatments  of  a  number  of  significant  topics 
in  cost-benefit  analysis,  and  presents  a  paradigm  for  carrying 
out  a  cost-benefit  study  as  well.  Topics  discussed  include 
decision  criteria,  the  identification  of  costs  and  benefits, 
quantifying  costs  and  benefits,  shadow  pricing,  the  discount  rote, 
social  and  environmental  impact  analysis  and  sensitivity  analy¬ 
sis.  The  steps  identified  by  the  authors  as  necessary  for 


carrying  out  a  cost-benefit  study  include  defining  the  problem 
designing  the  analysis,  data  collection,  qualitative  and  quan¬ 
titative  analyses,  and  preparation  of  a  final  report. 


Hishan,  E.  J.  Cost-Benefit  Analysis.  Praeger  Publishers, 
New  York:  1976.  This  monograph  is  perhaps  the  nost  extensive 
current  treatment  of  cost-benefit  analysis.  The  author  intro¬ 
duces  the  subject  via  discussion  of  a  series  of  recent  cost- 
benefit  studies,  including  studies  of  an  underground  railway 
and  of  a  tunnel  beneath  the  English  Channel.  Among  the  theo¬ 
retical  topics  treated  are  consumers'  surpius,  producers'  sur¬ 
plus,  shadow  pricing,  valuing  external  effects,  investment  cri¬ 


teria,  discount  rates  and  uncertainty.  The  concepts  of  com¬ 


pensating  and  equivalent  budget  variations  are  linked  to  the 


notion  of  consumers'  surplus. 


The  Economics  of  Public  Finance.  The  Brookings  Institute, 
Washington,  D.  C.:  1 9  7^*  •  Th  i  s  volume  is  a  collection  of  four 
essays  by  Blinder  and  Solow,  Bread,  Steiner,  and  Netzer.  Steiner's 
essay,  "Public  Expenditure  Budgeting",  is  intended  as  a  broad 
"statement  of  the  conceptual  issues  that  underlie  the  analysis 
of  the  political  -  economic  process  of  reaching  and  implementing 
public  expenditure  decisions."  He  distinguishes  between  "the 
theory  of  the  marginal  public  expenditure  .  .  .  (which)  takes 
as  given  the  legitimacy  of  government  activity,  and  is  concerned 
with  how  the  public  decision  maker  chooses  among  competing  demands 
for  his  limited  resources  .  .  .  and  the  theory  of  the  public  in¬ 
terest  .  .  .  (which)  concerns  the  way  in  which  demands  for  public 
activity  arise,  are  articulated,  and  are  legitimized."  The 
author  identifies  the  former  theory  with  cost-benefit  analysis 
and  proposes  a  sequential,  rather  than  simultaneous,  decision 
model  explicitly  considering  the  roles  of  Presidential  and  Con¬ 
gressional  actions. 


McKean,  R.  N.  Pubi i c  Spend i ng .  McGraw-Hill,  New  York: 
1968.  This  monograph  deals  with  both  the  positive  and  normative 
aspects  of  government  spending.  The  former  is  descriptive  while 
the  latter  is  prescriptive.  One  of  the  topics  in  the  latter 
realm  is  cost-benefit  analysis.  "Cost-benefit  analyses  are  at¬ 
tempts  to  estimate  certain  costs  and  gains  that  would  result 
from  alternative  courses  of  action  ....  The  term  .  .  .  was 
originally  associated  with  natural -resource  projects  but  has 
gradually  come  to  be  used  for  numerous  other  applications  .  .  . 
All  (cost-benefit)  analysis  involve  working  with  certain  common 
elements : 
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o  "objectives,  or  the  beneficial  things  to  be  achieved 

o  "alternatives,  or  the  possible  systems  or  arrange¬ 
ments  for  achieving  the  objectives 

o  "costs,  or  the  benefits  that  have  to  be  foregone 
if  one  of  the  alternatives  is  adopted 

o  "models,  or  the  sets  of  relationships  that  help 
one  trace  out  the  impacts  of  each  alternative  on 
achievements  (in  other  words,  on  benefits)  and 
costs 

o  "a  criterion,  involving  both  costs  and  benefits,  to 
identify  the  preferred  alternative." 

"...  cost-benefit  analysis  necessarily  involves  groping 
and  the  making  of  subjective  judgements,  .  .  .  (as  well  as)  an 
awareness  that  i ncommensurabl es  and  intangibles  are  pervasive." 


Baumol ,  W.  J.  Welfare  Economics  and  the  Theory  of  the  State. 
Harvard  University  Press,  Cambridge,  Massachusetts:  1965.  In  an 
extended  introduction,  the  author  summarizes  the  significant  ad¬ 
vances  in  welfare  economics  occurring  between  1950  and  1965. 

Among  these  are  the  welfare  applications  of  activity  analysis  by 
Koopmans  (1951,  1957),  Arrow  (1951)  and  Debreu  (1959);  the  direct 
application  of  welfare  economics  concepts  under  the  rubric  of  cost- 
benefit  analysis  by  Hitch  and  McKean  (195*0,  McKean  (1958),  Eckstein 
(1958),  Krutilla  and  Eckstein  (1958),  and  Hirshliefer,  DeHaven,  and 
Mil  liman  (i960);  the  refinement  of  the  concept  and  applications  of 
externalities  by  Meade  (1952),  Scitovsky  (195*0,  Buchanan  and 
Stubblebine  (1962),  Whinston  (1962),  Davis  and  Whinston  (1962) 
and  Coase  (i960);  and  further  developments  in  the  theory  of  the 
state  by  Marglin  (1963),  Baumol  (1963),  Downs  (1957),  Olson  (1965), 
and  Buchanan  and  Tullock  (1962). 

The  author's  development  of  a  theory  of  the  state  is  based 
on  "an  attempt  to  determine  which,  if  any,  arc  the  circumstances 
in  which  the  people  composing  an  economy  will  find  that  a  par¬ 
ticular  extension  of  the  authority  of  their  government  is  requi¬ 
site  for  the  most  efficient  pursuit  of  their  own  economic  interests." 
Baumol  employs  the  externality  argument  to  circumscribe,  in 
general  terms,  the  duties  most  efficiently  relegated  by  the  pri¬ 
vate  sector  to  government. 

McKean,  R.  N.  (cd.)  Issues  in  Defense  Economics.  Columbia 
University  Press,  New  YorlTi  1 967  •  This  volume  contains  eight 
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papers  dealing  with  the  general  topic  of  the  use  of  economic 
analysis  in  military  decision  making.  Malcolm  W.  Hoag  in 
"Increasing  Returns  in  Military  Production  Functions"  argues 
that  "prominent  production  economies  of  scale  do  apply  with 
special  frequency  in  military  applications,  and  that  they  can 
join  with  combat  conditions  to  create  increasing  returns  to 
scale  that  demand  wide-ranging  nonmarginal  analyses  of  military 
a) ternatives." 


Martin  J.  Bailey  in  "The  Market  Mechanism  in  the  Defense 
Department"  explores  the  history,  potential  use,  and  pros  and 
cons,  of  the  use  of  "transfer  prices"  within  the  military  to 
guide  resource  allocation. 


